DeclareDesignDeclareDesign basicsDeclareDesign deepdiveDeclareDesignDeclareDesignDeclareDesign BasicsHow to define and assess research designs
DeclareDesign: key resourcesModel: set of models of what causes what and howInquiry: a question stated in terms of the modelData strategy: the set of procedures we use to gather information from the world (sampling, assignment, measurement)Answer strategy: how we summarize the data produced by the data strategyDesign declaration is telling the computer (and readers) what M, I, D, and A are.
Design diagnosis is figuring out how the design will perform under imagined conditions.
Estimating “diagnosands” like power, bias, rmse, error rates, ethical harm, “amount learned”.
Diagnosis takes account of model uncertainty: it aims to identify models for which the design works well and models for which it does not
Redesign is the fine-tuning of features of the data- and answer strategies to understand how changing them affects the diagnosands
declare_model()
declare_inquiry()
declare_sampling()
declare_assignment()
declare_measurement()
declare_estimator()
and there are more declare_ functions!
draw_data(design)draw_estimands(design)draw_estimates(design)get_estimates(design, data)run_design(design), simulate_design(design)diagnose_design(design)redesign(design, N = 200)compare_designs(), compare_diagnoses()https://raw.githubusercontent.com/rstudio/cheatsheets/master/declaredesign.pdf
?DeclareDesign+designEach step is a function (or rather: a function that generates functions) and each function presupposes what is created by previous functions.
main data frame in and push the main dataframe out; this data frame normally builds up as you move along the pipe.Each step is a function (or rather: a function that generates functions) and each function presupposes what is created by previous functions.
declare_estimator steps take the main data frame in and send out an estimator_df dataframedeclare_inquiry steps take the main data frame in and send out an estimand_df dataframe.You can also just run through the whole design once by typing the name of the design:
Research design declaration summary
Step 1 (model): declare_model(N = 100, Y = rnorm(N, mean)) ---------------------
Step 2 (inquiry): declare_inquiry(Q = 0) ---------------------------------------
Step 3 (estimator): declare_estimator(Y ~ 1) -----------------------------------
Run of the design:
inquiry estimand estimator term estimate std.error statistic p.value
Q 0 estimator (Intercept) 0.155 0.108 1.43 0.155
conf.low conf.high df outcome
-0.0597 0.371 99 Y
Or by asking for a run of the design
one_run <- simplest_design |> run_design()
one_run |> kable(digits = 2) |> kable_styling(font_size = 18)| inquiry | estimand | estimator | term | estimate | std.error | statistic | p.value | conf.low | conf.high | df | outcome |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Q | 0 | estimator | (Intercept) | 0.08 | 0.1 | 0.8 | 0.43 | -0.12 | 0.28 | 99 | Y |
A single run creates data, calculates estimands (the answer to inquiries) and calculates estimates plus ancillary statistics.
Or by asking for many runs of the design
some_runs <- simplest_design |> simulate_design(sims = 1000)
some_runs |> kable(digits = 2) |> kable_styling(font_size = 16)| design | sim_ID | inquiry | estimand | estimator | term | estimate | std.error | statistic | p.value | conf.low | conf.high | df | outcome |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| simplest_design | 1 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.32 | 0.75 | -0.15 | 0.21 | 99 | Y |
| simplest_design | 2 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.03 | 0.98 | -0.19 | 0.20 | 99 | Y |
| simplest_design | 3 | Q | 0 | estimator | (Intercept) | -0.13 | 0.09 | -1.46 | 0.15 | -0.31 | 0.05 | 99 | Y |
| simplest_design | 4 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.36 | 0.18 | -0.35 | 0.06 | 99 | Y |
| simplest_design | 5 | Q | 0 | estimator | (Intercept) | 0.02 | 0.12 | 0.16 | 0.88 | -0.22 | 0.26 | 99 | Y |
| simplest_design | 6 | Q | 0 | estimator | (Intercept) | -0.07 | 0.09 | -0.69 | 0.49 | -0.25 | 0.12 | 99 | Y |
| simplest_design | 7 | Q | 0 | estimator | (Intercept) | -0.09 | 0.09 | -0.99 | 0.33 | -0.28 | 0.09 | 99 | Y |
| simplest_design | 8 | Q | 0 | estimator | (Intercept) | -0.10 | 0.09 | -1.16 | 0.25 | -0.27 | 0.07 | 99 | Y |
| simplest_design | 9 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.12 | 0.90 | -0.22 | 0.19 | 99 | Y |
| simplest_design | 10 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.14 | 0.89 | -0.21 | 0.24 | 99 | Y |
| simplest_design | 11 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.70 | 0.49 | -0.25 | 0.12 | 99 | Y |
| simplest_design | 12 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.83 | 0.41 | -0.11 | 0.28 | 99 | Y |
| simplest_design | 13 | Q | 0 | estimator | (Intercept) | -0.26 | 0.10 | -2.52 | 0.01 | -0.46 | -0.06 | 99 | Y |
| simplest_design | 14 | Q | 0 | estimator | (Intercept) | 0.08 | 0.09 | 0.81 | 0.42 | -0.11 | 0.26 | 99 | Y |
| simplest_design | 15 | Q | 0 | estimator | (Intercept) | 0.07 | 0.09 | 0.72 | 0.47 | -0.12 | 0.25 | 99 | Y |
| simplest_design | 16 | Q | 0 | estimator | (Intercept) | 0.15 | 0.10 | 1.49 | 0.14 | -0.05 | 0.35 | 99 | Y |
| simplest_design | 17 | Q | 0 | estimator | (Intercept) | -0.14 | 0.09 | -1.54 | 0.13 | -0.32 | 0.04 | 99 | Y |
| simplest_design | 18 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -0.98 | 0.33 | -0.30 | 0.10 | 99 | Y |
| simplest_design | 19 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.62 | 0.54 | -0.13 | 0.24 | 99 | Y |
| simplest_design | 20 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.29 | 0.77 | -0.15 | 0.20 | 99 | Y |
| simplest_design | 21 | Q | 0 | estimator | (Intercept) | -0.08 | 0.09 | -0.93 | 0.35 | -0.27 | 0.10 | 99 | Y |
| simplest_design | 22 | Q | 0 | estimator | (Intercept) | -0.09 | 0.11 | -0.81 | 0.42 | -0.29 | 0.12 | 99 | Y |
| simplest_design | 23 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.06 | 0.29 | -0.31 | 0.10 | 99 | Y |
| simplest_design | 24 | Q | 0 | estimator | (Intercept) | 0.10 | 0.09 | 1.07 | 0.29 | -0.08 | 0.28 | 99 | Y |
| simplest_design | 25 | Q | 0 | estimator | (Intercept) | 0.01 | 0.09 | 0.16 | 0.88 | -0.17 | 0.20 | 99 | Y |
| simplest_design | 26 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.27 | 0.79 | -0.23 | 0.18 | 99 | Y |
| simplest_design | 27 | Q | 0 | estimator | (Intercept) | 0.22 | 0.10 | 2.17 | 0.03 | 0.02 | 0.42 | 99 | Y |
| simplest_design | 28 | Q | 0 | estimator | (Intercept) | -0.35 | 0.10 | -3.35 | 0.00 | -0.55 | -0.14 | 99 | Y |
| simplest_design | 29 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.39 | 0.69 | -0.22 | 0.15 | 99 | Y |
| simplest_design | 30 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.25 | 0.81 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 31 | Q | 0 | estimator | (Intercept) | 0.21 | 0.10 | 2.05 | 0.04 | 0.01 | 0.40 | 99 | Y |
| simplest_design | 32 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.09 | 0.93 | -0.20 | 0.18 | 99 | Y |
| simplest_design | 33 | Q | 0 | estimator | (Intercept) | 0.15 | 0.09 | 1.64 | 0.10 | -0.03 | 0.34 | 99 | Y |
| simplest_design | 34 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.42 | 0.68 | -0.25 | 0.16 | 99 | Y |
| simplest_design | 35 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.53 | 0.60 | -0.13 | 0.23 | 99 | Y |
| simplest_design | 36 | Q | 0 | estimator | (Intercept) | -0.15 | 0.11 | -1.35 | 0.18 | -0.36 | 0.07 | 99 | Y |
| simplest_design | 37 | Q | 0 | estimator | (Intercept) | 0.14 | 0.11 | 1.27 | 0.21 | -0.08 | 0.35 | 99 | Y |
| simplest_design | 38 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.26 | 0.80 | -0.18 | 0.23 | 99 | Y |
| simplest_design | 39 | Q | 0 | estimator | (Intercept) | 0.22 | 0.09 | 2.38 | 0.02 | 0.04 | 0.40 | 99 | Y |
| simplest_design | 40 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.43 | 0.67 | -0.26 | 0.17 | 99 | Y |
| simplest_design | 41 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.06 | 0.95 | -0.19 | 0.18 | 99 | Y |
| simplest_design | 42 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.14 | 0.89 | -0.21 | 0.18 | 99 | Y |
| simplest_design | 43 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.23 | 0.22 | -0.33 | 0.08 | 99 | Y |
| simplest_design | 44 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.05 | 0.96 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 45 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.48 | 0.63 | -0.15 | 0.24 | 99 | Y |
| simplest_design | 46 | Q | 0 | estimator | (Intercept) | 0.04 | 0.11 | 0.39 | 0.70 | -0.17 | 0.26 | 99 | Y |
| simplest_design | 47 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.58 | 0.56 | -0.15 | 0.28 | 99 | Y |
| simplest_design | 48 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.52 | 0.13 | -0.36 | 0.05 | 99 | Y |
| simplest_design | 49 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.28 | 0.78 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 50 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.80 | 0.42 | -0.28 | 0.12 | 99 | Y |
| simplest_design | 51 | Q | 0 | estimator | (Intercept) | -0.07 | 0.11 | -0.64 | 0.52 | -0.29 | 0.15 | 99 | Y |
| simplest_design | 52 | Q | 0 | estimator | (Intercept) | 0.09 | 0.11 | 0.80 | 0.43 | -0.13 | 0.30 | 99 | Y |
| simplest_design | 53 | Q | 0 | estimator | (Intercept) | -0.10 | 0.11 | -0.96 | 0.34 | -0.31 | 0.11 | 99 | Y |
| simplest_design | 54 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.42 | 0.68 | -0.22 | 0.14 | 99 | Y |
| simplest_design | 55 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.17 | 0.25 | -0.08 | 0.31 | 99 | Y |
| simplest_design | 56 | Q | 0 | estimator | (Intercept) | -0.02 | 0.11 | -0.16 | 0.88 | -0.23 | 0.19 | 99 | Y |
| simplest_design | 57 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.12 | 0.26 | -0.31 | 0.09 | 99 | Y |
| simplest_design | 58 | Q | 0 | estimator | (Intercept) | 0.12 | 0.11 | 1.06 | 0.29 | -0.10 | 0.33 | 99 | Y |
| simplest_design | 59 | Q | 0 | estimator | (Intercept) | 0.17 | 0.10 | 1.67 | 0.10 | -0.03 | 0.37 | 99 | Y |
| simplest_design | 60 | Q | 0 | estimator | (Intercept) | -0.10 | 0.09 | -1.15 | 0.25 | -0.28 | 0.08 | 99 | Y |
| simplest_design | 61 | Q | 0 | estimator | (Intercept) | 0.16 | 0.10 | 1.68 | 0.10 | -0.03 | 0.35 | 99 | Y |
| simplest_design | 62 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.84 | 0.40 | -0.11 | 0.27 | 99 | Y |
| simplest_design | 63 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.45 | 0.66 | -0.25 | 0.16 | 99 | Y |
| simplest_design | 64 | Q | 0 | estimator | (Intercept) | 0.17 | 0.11 | 1.58 | 0.12 | -0.04 | 0.38 | 99 | Y |
| simplest_design | 65 | Q | 0 | estimator | (Intercept) | -0.14 | 0.11 | -1.26 | 0.21 | -0.36 | 0.08 | 99 | Y |
| simplest_design | 66 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.61 | 0.54 | -0.13 | 0.25 | 99 | Y |
| simplest_design | 67 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.52 | 0.60 | -0.24 | 0.14 | 99 | Y |
| simplest_design | 68 | Q | 0 | estimator | (Intercept) | -0.08 | 0.09 | -0.83 | 0.41 | -0.26 | 0.11 | 99 | Y |
| simplest_design | 69 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.42 | 0.68 | -0.16 | 0.24 | 99 | Y |
| simplest_design | 70 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | 0.02 | 0.99 | -0.17 | 0.18 | 99 | Y |
| simplest_design | 71 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.49 | 0.63 | -0.23 | 0.14 | 99 | Y |
| simplest_design | 72 | Q | 0 | estimator | (Intercept) | 0.13 | 0.11 | 1.14 | 0.26 | -0.09 | 0.35 | 99 | Y |
| simplest_design | 73 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.38 | 0.71 | -0.15 | 0.21 | 99 | Y |
| simplest_design | 74 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.45 | 0.66 | -0.16 | 0.25 | 99 | Y |
| simplest_design | 75 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.56 | 0.58 | -0.26 | 0.15 | 99 | Y |
| simplest_design | 76 | Q | 0 | estimator | (Intercept) | -0.15 | 0.10 | -1.54 | 0.13 | -0.34 | 0.04 | 99 | Y |
| simplest_design | 77 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.58 | 0.56 | -0.28 | 0.15 | 99 | Y |
| simplest_design | 78 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.25 | 0.80 | -0.20 | 0.16 | 99 | Y |
| simplest_design | 79 | Q | 0 | estimator | (Intercept) | 0.12 | 0.11 | 1.02 | 0.31 | -0.11 | 0.34 | 99 | Y |
| simplest_design | 80 | Q | 0 | estimator | (Intercept) | -0.07 | 0.11 | -0.68 | 0.50 | -0.28 | 0.14 | 99 | Y |
| simplest_design | 81 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.21 | 0.84 | -0.20 | 0.17 | 99 | Y |
| simplest_design | 82 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | -0.04 | 0.97 | -0.22 | 0.21 | 99 | Y |
| simplest_design | 83 | Q | 0 | estimator | (Intercept) | -0.11 | 0.11 | -1.03 | 0.31 | -0.33 | 0.11 | 99 | Y |
| simplest_design | 84 | Q | 0 | estimator | (Intercept) | -0.11 | 0.11 | -1.00 | 0.32 | -0.33 | 0.11 | 99 | Y |
| simplest_design | 85 | Q | 0 | estimator | (Intercept) | -0.02 | 0.11 | -0.18 | 0.86 | -0.24 | 0.20 | 99 | Y |
| simplest_design | 86 | Q | 0 | estimator | (Intercept) | 0.17 | 0.08 | 2.00 | 0.05 | 0.00 | 0.33 | 99 | Y |
| simplest_design | 87 | Q | 0 | estimator | (Intercept) | -0.17 | 0.11 | -1.55 | 0.13 | -0.38 | 0.05 | 99 | Y |
| simplest_design | 88 | Q | 0 | estimator | (Intercept) | -0.23 | 0.11 | -2.02 | 0.05 | -0.45 | 0.00 | 99 | Y |
| simplest_design | 89 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.62 | 0.54 | -0.23 | 0.12 | 99 | Y |
| simplest_design | 90 | Q | 0 | estimator | (Intercept) | 0.16 | 0.09 | 1.78 | 0.08 | -0.02 | 0.33 | 99 | Y |
| simplest_design | 91 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.18 | 0.24 | -0.08 | 0.33 | 99 | Y |
| simplest_design | 92 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.22 | 0.22 | -0.08 | 0.33 | 99 | Y |
| simplest_design | 93 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.69 | 0.49 | -0.24 | 0.12 | 99 | Y |
| simplest_design | 94 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.03 | 0.30 | -0.09 | 0.29 | 99 | Y |
| simplest_design | 95 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 0.98 | 0.33 | -0.10 | 0.31 | 99 | Y |
| simplest_design | 96 | Q | 0 | estimator | (Intercept) | -0.21 | 0.10 | -2.16 | 0.03 | -0.40 | -0.02 | 99 | Y |
| simplest_design | 97 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.01 | 1.00 | -0.20 | 0.20 | 99 | Y |
| simplest_design | 98 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.37 | 0.72 | -0.14 | 0.21 | 99 | Y |
| simplest_design | 99 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.09 | 0.93 | -0.21 | 0.19 | 99 | Y |
| simplest_design | 100 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.35 | 0.73 | -0.22 | 0.15 | 99 | Y |
| simplest_design | 101 | Q | 0 | estimator | (Intercept) | -0.15 | 0.11 | -1.38 | 0.17 | -0.36 | 0.06 | 99 | Y |
| simplest_design | 102 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.63 | 0.53 | -0.13 | 0.24 | 99 | Y |
| simplest_design | 103 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | -0.02 | 0.98 | -0.21 | 0.20 | 99 | Y |
| simplest_design | 104 | Q | 0 | estimator | (Intercept) | -0.24 | 0.10 | -2.29 | 0.02 | -0.44 | -0.03 | 99 | Y |
| simplest_design | 105 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.51 | 0.61 | -0.27 | 0.16 | 99 | Y |
| simplest_design | 106 | Q | 0 | estimator | (Intercept) | -0.16 | 0.09 | -1.71 | 0.09 | -0.35 | 0.03 | 99 | Y |
| simplest_design | 107 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.11 | 0.27 | -0.31 | 0.09 | 99 | Y |
| simplest_design | 108 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.18 | 0.86 | -0.19 | 0.23 | 99 | Y |
| simplest_design | 109 | Q | 0 | estimator | (Intercept) | 0.07 | 0.09 | 0.77 | 0.45 | -0.11 | 0.26 | 99 | Y |
| simplest_design | 110 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.30 | 0.77 | -0.17 | 0.23 | 99 | Y |
| simplest_design | 111 | Q | 0 | estimator | (Intercept) | 0.25 | 0.09 | 2.82 | 0.01 | 0.08 | 0.43 | 99 | Y |
| simplest_design | 112 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.25 | 0.80 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 113 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.13 | 0.90 | -0.21 | 0.24 | 99 | Y |
| simplest_design | 114 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.54 | 0.59 | -0.25 | 0.14 | 99 | Y |
| simplest_design | 115 | Q | 0 | estimator | (Intercept) | -0.17 | 0.09 | -1.87 | 0.06 | -0.35 | 0.01 | 99 | Y |
| simplest_design | 116 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.21 | 0.83 | -0.18 | 0.22 | 99 | Y |
| simplest_design | 117 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -0.96 | 0.34 | -0.30 | 0.10 | 99 | Y |
| simplest_design | 118 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.39 | 0.17 | -0.34 | 0.06 | 99 | Y |
| simplest_design | 119 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.95 | 0.34 | -0.28 | 0.10 | 99 | Y |
| simplest_design | 120 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.70 | 0.48 | -0.12 | 0.25 | 99 | Y |
| simplest_design | 121 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.09 | 0.93 | -0.19 | 0.20 | 99 | Y |
| simplest_design | 122 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.14 | 0.89 | -0.18 | 0.21 | 99 | Y |
| simplest_design | 123 | Q | 0 | estimator | (Intercept) | 0.25 | 0.09 | 2.68 | 0.01 | 0.06 | 0.44 | 99 | Y |
| simplest_design | 124 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.73 | 0.46 | -0.13 | 0.27 | 99 | Y |
| simplest_design | 125 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.46 | 0.15 | -0.33 | 0.05 | 99 | Y |
| simplest_design | 126 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.73 | 0.47 | -0.27 | 0.13 | 99 | Y |
| simplest_design | 127 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.82 | 0.07 | -0.38 | 0.02 | 99 | Y |
| simplest_design | 128 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.25 | 0.80 | -0.18 | 0.23 | 99 | Y |
| simplest_design | 129 | Q | 0 | estimator | (Intercept) | -0.10 | 0.09 | -1.07 | 0.29 | -0.28 | 0.08 | 99 | Y |
| simplest_design | 130 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.23 | 0.82 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 131 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | -0.03 | 0.98 | -0.22 | 0.21 | 99 | Y |
| simplest_design | 132 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.49 | 0.62 | -0.26 | 0.16 | 99 | Y |
| simplest_design | 133 | Q | 0 | estimator | (Intercept) | 0.17 | 0.09 | 1.83 | 0.07 | -0.01 | 0.35 | 99 | Y |
| simplest_design | 134 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.11 | 0.27 | -0.30 | 0.08 | 99 | Y |
| simplest_design | 135 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.33 | 0.19 | -0.06 | 0.32 | 99 | Y |
| simplest_design | 136 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.46 | 0.65 | -0.26 | 0.16 | 99 | Y |
| simplest_design | 137 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.50 | 0.62 | -0.23 | 0.14 | 99 | Y |
| simplest_design | 138 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.16 | 0.25 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 139 | Q | 0 | estimator | (Intercept) | -0.12 | 0.09 | -1.32 | 0.19 | -0.31 | 0.06 | 99 | Y |
| simplest_design | 140 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.66 | 0.51 | -0.26 | 0.13 | 99 | Y |
| simplest_design | 141 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.64 | 0.53 | -0.12 | 0.22 | 99 | Y |
| simplest_design | 142 | Q | 0 | estimator | (Intercept) | 0.00 | 0.12 | -0.04 | 0.97 | -0.24 | 0.23 | 99 | Y |
| simplest_design | 143 | Q | 0 | estimator | (Intercept) | 0.04 | 0.09 | 0.44 | 0.66 | -0.14 | 0.22 | 99 | Y |
| simplest_design | 144 | Q | 0 | estimator | (Intercept) | -0.01 | 0.11 | -0.08 | 0.94 | -0.23 | 0.21 | 99 | Y |
| simplest_design | 145 | Q | 0 | estimator | (Intercept) | -0.05 | 0.12 | -0.43 | 0.67 | -0.28 | 0.18 | 99 | Y |
| simplest_design | 146 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.02 | 0.31 | -0.31 | 0.10 | 99 | Y |
| simplest_design | 147 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.28 | 0.78 | -0.17 | 0.23 | 99 | Y |
| simplest_design | 148 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.34 | 0.74 | -0.22 | 0.16 | 99 | Y |
| simplest_design | 149 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.53 | 0.59 | -0.26 | 0.15 | 99 | Y |
| simplest_design | 150 | Q | 0 | estimator | (Intercept) | -0.19 | 0.10 | -1.88 | 0.06 | -0.39 | 0.01 | 99 | Y |
| simplest_design | 151 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.77 | 0.44 | -0.12 | 0.28 | 99 | Y |
| simplest_design | 152 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.16 | 0.88 | -0.20 | 0.23 | 99 | Y |
| simplest_design | 153 | Q | 0 | estimator | (Intercept) | -0.15 | 0.09 | -1.58 | 0.12 | -0.33 | 0.04 | 99 | Y |
| simplest_design | 154 | Q | 0 | estimator | (Intercept) | -0.18 | 0.09 | -1.94 | 0.06 | -0.37 | 0.00 | 99 | Y |
| simplest_design | 155 | Q | 0 | estimator | (Intercept) | 0.12 | 0.11 | 1.09 | 0.28 | -0.10 | 0.33 | 99 | Y |
| simplest_design | 156 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.72 | 0.09 | -0.38 | 0.03 | 99 | Y |
| simplest_design | 157 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.37 | 0.71 | -0.26 | 0.18 | 99 | Y |
| simplest_design | 158 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.07 | 0.94 | -0.21 | 0.22 | 99 | Y |
| simplest_design | 159 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.89 | 0.38 | -0.27 | 0.10 | 99 | Y |
| simplest_design | 160 | Q | 0 | estimator | (Intercept) | -0.19 | 0.10 | -1.92 | 0.06 | -0.39 | 0.01 | 99 | Y |
| simplest_design | 161 | Q | 0 | estimator | (Intercept) | 0.17 | 0.10 | 1.79 | 0.08 | -0.02 | 0.36 | 99 | Y |
| simplest_design | 162 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.72 | 0.47 | -0.12 | 0.26 | 99 | Y |
| simplest_design | 163 | Q | 0 | estimator | (Intercept) | 0.12 | 0.12 | 1.04 | 0.30 | -0.11 | 0.35 | 99 | Y |
| simplest_design | 164 | Q | 0 | estimator | (Intercept) | 0.13 | 0.09 | 1.44 | 0.15 | -0.05 | 0.32 | 99 | Y |
| simplest_design | 165 | Q | 0 | estimator | (Intercept) | 0.08 | 0.09 | 0.98 | 0.33 | -0.09 | 0.25 | 99 | Y |
| simplest_design | 166 | Q | 0 | estimator | (Intercept) | 0.23 | 0.11 | 2.12 | 0.04 | 0.01 | 0.44 | 99 | Y |
| simplest_design | 167 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.28 | 0.78 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 168 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.88 | 0.38 | -0.12 | 0.30 | 99 | Y |
| simplest_design | 169 | Q | 0 | estimator | (Intercept) | 0.04 | 0.11 | 0.33 | 0.74 | -0.19 | 0.26 | 99 | Y |
| simplest_design | 170 | Q | 0 | estimator | (Intercept) | 0.12 | 0.09 | 1.32 | 0.19 | -0.06 | 0.29 | 99 | Y |
| simplest_design | 171 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.34 | 0.74 | -0.20 | 0.14 | 99 | Y |
| simplest_design | 172 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.69 | 0.49 | -0.13 | 0.27 | 99 | Y |
| simplest_design | 173 | Q | 0 | estimator | (Intercept) | -0.06 | 0.08 | -0.73 | 0.47 | -0.23 | 0.10 | 99 | Y |
| simplest_design | 174 | Q | 0 | estimator | (Intercept) | 0.01 | 0.09 | 0.14 | 0.89 | -0.17 | 0.20 | 99 | Y |
| simplest_design | 175 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.88 | 0.38 | -0.29 | 0.11 | 99 | Y |
| simplest_design | 176 | Q | 0 | estimator | (Intercept) | 0.18 | 0.11 | 1.62 | 0.11 | -0.04 | 0.39 | 99 | Y |
| simplest_design | 177 | Q | 0 | estimator | (Intercept) | -0.12 | 0.11 | -1.14 | 0.26 | -0.33 | 0.09 | 99 | Y |
| simplest_design | 178 | Q | 0 | estimator | (Intercept) | -0.14 | 0.11 | -1.23 | 0.22 | -0.36 | 0.09 | 99 | Y |
| simplest_design | 179 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.71 | 0.48 | -0.24 | 0.11 | 99 | Y |
| simplest_design | 180 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.41 | 0.68 | -0.27 | 0.18 | 99 | Y |
| simplest_design | 181 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.40 | 0.69 | -0.25 | 0.17 | 99 | Y |
| simplest_design | 182 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.32 | 0.75 | -0.23 | 0.16 | 99 | Y |
| simplest_design | 183 | Q | 0 | estimator | (Intercept) | 0.14 | 0.10 | 1.37 | 0.17 | -0.06 | 0.35 | 99 | Y |
| simplest_design | 184 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.14 | 0.89 | -0.20 | 0.23 | 99 | Y |
| simplest_design | 185 | Q | 0 | estimator | (Intercept) | 0.11 | 0.11 | 0.98 | 0.33 | -0.12 | 0.34 | 99 | Y |
| simplest_design | 186 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.41 | 0.68 | -0.24 | 0.16 | 99 | Y |
| simplest_design | 187 | Q | 0 | estimator | (Intercept) | 0.17 | 0.10 | 1.63 | 0.11 | -0.04 | 0.37 | 99 | Y |
| simplest_design | 188 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.53 | 0.60 | -0.14 | 0.24 | 99 | Y |
| simplest_design | 189 | Q | 0 | estimator | (Intercept) | 0.11 | 0.11 | 1.03 | 0.31 | -0.10 | 0.32 | 99 | Y |
| simplest_design | 190 | Q | 0 | estimator | (Intercept) | 0.00 | 0.08 | -0.01 | 0.99 | -0.17 | 0.17 | 99 | Y |
| simplest_design | 191 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.44 | 0.66 | -0.15 | 0.24 | 99 | Y |
| simplest_design | 192 | Q | 0 | estimator | (Intercept) | 0.17 | 0.10 | 1.60 | 0.11 | -0.04 | 0.37 | 99 | Y |
| simplest_design | 193 | Q | 0 | estimator | (Intercept) | -0.09 | 0.09 | -1.09 | 0.28 | -0.27 | 0.08 | 99 | Y |
| simplest_design | 194 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.60 | 0.55 | -0.24 | 0.13 | 99 | Y |
| simplest_design | 195 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.23 | 0.82 | -0.22 | 0.17 | 99 | Y |
| simplest_design | 196 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.28 | 0.78 | -0.26 | 0.19 | 99 | Y |
| simplest_design | 197 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.54 | 0.59 | -0.16 | 0.28 | 99 | Y |
| simplest_design | 198 | Q | 0 | estimator | (Intercept) | -0.07 | 0.09 | -0.78 | 0.44 | -0.25 | 0.11 | 99 | Y |
| simplest_design | 199 | Q | 0 | estimator | (Intercept) | 0.18 | 0.09 | 1.94 | 0.05 | 0.00 | 0.37 | 99 | Y |
| simplest_design | 200 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.75 | 0.46 | -0.11 | 0.23 | 99 | Y |
| simplest_design | 201 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.53 | 0.59 | -0.15 | 0.26 | 99 | Y |
| simplest_design | 202 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.30 | 0.77 | -0.15 | 0.20 | 99 | Y |
| simplest_design | 203 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.27 | 0.79 | -0.16 | 0.21 | 99 | Y |
| simplest_design | 204 | Q | 0 | estimator | (Intercept) | -0.09 | 0.09 | -1.04 | 0.30 | -0.27 | 0.09 | 99 | Y |
| simplest_design | 205 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.46 | 0.65 | -0.26 | 0.16 | 99 | Y |
| simplest_design | 206 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.44 | 0.66 | -0.23 | 0.15 | 99 | Y |
| simplest_design | 207 | Q | 0 | estimator | (Intercept) | -0.08 | 0.09 | -0.84 | 0.40 | -0.25 | 0.10 | 99 | Y |
| simplest_design | 208 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.34 | 0.18 | -0.35 | 0.07 | 99 | Y |
| simplest_design | 209 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.41 | 0.68 | -0.24 | 0.16 | 99 | Y |
| simplest_design | 210 | Q | 0 | estimator | (Intercept) | 0.12 | 0.11 | 1.02 | 0.31 | -0.11 | 0.34 | 99 | Y |
| simplest_design | 211 | Q | 0 | estimator | (Intercept) | -0.02 | 0.11 | -0.19 | 0.85 | -0.24 | 0.19 | 99 | Y |
| simplest_design | 212 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | 0.03 | 0.98 | -0.17 | 0.18 | 99 | Y |
| simplest_design | 213 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.14 | 0.26 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 214 | Q | 0 | estimator | (Intercept) | 0.01 | 0.09 | 0.06 | 0.95 | -0.18 | 0.19 | 99 | Y |
| simplest_design | 215 | Q | 0 | estimator | (Intercept) | 0.10 | 0.11 | 0.91 | 0.36 | -0.12 | 0.32 | 99 | Y |
| simplest_design | 216 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.19 | 0.85 | -0.22 | 0.19 | 99 | Y |
| simplest_design | 217 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.54 | 0.59 | -0.26 | 0.15 | 99 | Y |
| simplest_design | 218 | Q | 0 | estimator | (Intercept) | -0.16 | 0.09 | -1.77 | 0.08 | -0.33 | 0.02 | 99 | Y |
| simplest_design | 219 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.54 | 0.59 | -0.14 | 0.25 | 99 | Y |
| simplest_design | 220 | Q | 0 | estimator | (Intercept) | 0.13 | 0.09 | 1.50 | 0.14 | -0.04 | 0.31 | 99 | Y |
| simplest_design | 221 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.10 | 0.92 | -0.20 | 0.18 | 99 | Y |
| simplest_design | 222 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.05 | 0.30 | -0.09 | 0.29 | 99 | Y |
| simplest_design | 223 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.79 | 0.43 | -0.28 | 0.12 | 99 | Y |
| simplest_design | 224 | Q | 0 | estimator | (Intercept) | -0.13 | 0.11 | -1.17 | 0.24 | -0.35 | 0.09 | 99 | Y |
| simplest_design | 225 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.26 | 0.21 | -0.07 | 0.33 | 99 | Y |
| simplest_design | 226 | Q | 0 | estimator | (Intercept) | -0.14 | 0.11 | -1.27 | 0.21 | -0.36 | 0.08 | 99 | Y |
| simplest_design | 227 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.26 | 0.21 | -0.33 | 0.07 | 99 | Y |
| simplest_design | 228 | Q | 0 | estimator | (Intercept) | -0.20 | 0.11 | -1.90 | 0.06 | -0.41 | 0.01 | 99 | Y |
| simplest_design | 229 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.06 | 0.96 | -0.19 | 0.18 | 99 | Y |
| simplest_design | 230 | Q | 0 | estimator | (Intercept) | 0.21 | 0.10 | 2.07 | 0.04 | 0.01 | 0.42 | 99 | Y |
| simplest_design | 231 | Q | 0 | estimator | (Intercept) | -0.21 | 0.10 | -2.03 | 0.05 | -0.42 | 0.00 | 99 | Y |
| simplest_design | 232 | Q | 0 | estimator | (Intercept) | 0.12 | 0.09 | 1.35 | 0.18 | -0.06 | 0.30 | 99 | Y |
| simplest_design | 233 | Q | 0 | estimator | (Intercept) | -0.12 | 0.11 | -1.14 | 0.26 | -0.33 | 0.09 | 99 | Y |
| simplest_design | 234 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.20 | 0.84 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 235 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.11 | 0.91 | -0.18 | 0.20 | 99 | Y |
| simplest_design | 236 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.54 | 0.59 | -0.24 | 0.14 | 99 | Y |
| simplest_design | 237 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.22 | 0.22 | -0.31 | 0.07 | 99 | Y |
| simplest_design | 238 | Q | 0 | estimator | (Intercept) | 0.10 | 0.09 | 1.12 | 0.26 | -0.08 | 0.29 | 99 | Y |
| simplest_design | 239 | Q | 0 | estimator | (Intercept) | 0.15 | 0.09 | 1.68 | 0.10 | -0.03 | 0.34 | 99 | Y |
| simplest_design | 240 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.45 | 0.65 | -0.15 | 0.23 | 99 | Y |
| simplest_design | 241 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | 0.00 | 1.00 | -0.21 | 0.21 | 99 | Y |
| simplest_design | 242 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.33 | 0.74 | -0.17 | 0.24 | 99 | Y |
| simplest_design | 243 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.44 | 0.15 | -0.34 | 0.05 | 99 | Y |
| simplest_design | 244 | Q | 0 | estimator | (Intercept) | 0.10 | 0.09 | 1.07 | 0.29 | -0.09 | 0.29 | 99 | Y |
| simplest_design | 245 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.14 | 0.89 | -0.21 | 0.18 | 99 | Y |
| simplest_design | 246 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.26 | 0.80 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 247 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.23 | 0.22 | -0.32 | 0.08 | 99 | Y |
| simplest_design | 248 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 0.95 | 0.35 | -0.10 | 0.27 | 99 | Y |
| simplest_design | 249 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.32 | 0.75 | -0.17 | 0.24 | 99 | Y |
| simplest_design | 250 | Q | 0 | estimator | (Intercept) | 0.18 | 0.09 | 1.93 | 0.06 | -0.01 | 0.36 | 99 | Y |
| simplest_design | 251 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.06 | 0.96 | -0.19 | 0.18 | 99 | Y |
| simplest_design | 252 | Q | 0 | estimator | (Intercept) | 0.08 | 0.09 | 0.92 | 0.36 | -0.09 | 0.25 | 99 | Y |
| simplest_design | 253 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.13 | 0.90 | -0.19 | 0.22 | 99 | Y |
| simplest_design | 254 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 0.96 | 0.34 | -0.09 | 0.26 | 99 | Y |
| simplest_design | 255 | Q | 0 | estimator | (Intercept) | -0.15 | 0.10 | -1.44 | 0.15 | -0.35 | 0.05 | 99 | Y |
| simplest_design | 256 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.14 | 0.26 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 257 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.33 | 0.74 | -0.15 | 0.21 | 99 | Y |
| simplest_design | 258 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.40 | 0.69 | -0.23 | 0.15 | 99 | Y |
| simplest_design | 259 | Q | 0 | estimator | (Intercept) | 0.22 | 0.10 | 2.15 | 0.03 | 0.02 | 0.42 | 99 | Y |
| simplest_design | 260 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.06 | 0.95 | -0.21 | 0.20 | 99 | Y |
| simplest_design | 261 | Q | 0 | estimator | (Intercept) | 0.18 | 0.10 | 1.75 | 0.08 | -0.02 | 0.39 | 99 | Y |
| simplest_design | 262 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.20 | 0.85 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 263 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.50 | 0.61 | -0.14 | 0.24 | 99 | Y |
| simplest_design | 264 | Q | 0 | estimator | (Intercept) | 0.05 | 0.12 | 0.45 | 0.66 | -0.18 | 0.28 | 99 | Y |
| simplest_design | 265 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.21 | 0.84 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 266 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.23 | 0.82 | -0.19 | 0.24 | 99 | Y |
| simplest_design | 267 | Q | 0 | estimator | (Intercept) | 0.02 | 0.08 | 0.19 | 0.85 | -0.15 | 0.18 | 99 | Y |
| simplest_design | 268 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.08 | 0.94 | -0.20 | 0.22 | 99 | Y |
| simplest_design | 269 | Q | 0 | estimator | (Intercept) | -0.28 | 0.10 | -2.72 | 0.01 | -0.48 | -0.07 | 99 | Y |
| simplest_design | 270 | Q | 0 | estimator | (Intercept) | 0.08 | 0.11 | 0.80 | 0.42 | -0.12 | 0.29 | 99 | Y |
| simplest_design | 271 | Q | 0 | estimator | (Intercept) | 0.08 | 0.09 | 0.90 | 0.37 | -0.09 | 0.25 | 99 | Y |
| simplest_design | 272 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.47 | 0.64 | -0.15 | 0.24 | 99 | Y |
| simplest_design | 273 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 0.99 | 0.32 | -0.09 | 0.27 | 99 | Y |
| simplest_design | 274 | Q | 0 | estimator | (Intercept) | -0.24 | 0.11 | -2.15 | 0.03 | -0.46 | -0.02 | 99 | Y |
| simplest_design | 275 | Q | 0 | estimator | (Intercept) | 0.05 | 0.11 | 0.49 | 0.62 | -0.16 | 0.27 | 99 | Y |
| simplest_design | 276 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.26 | 0.21 | -0.33 | 0.07 | 99 | Y |
| simplest_design | 277 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.40 | 0.69 | -0.22 | 0.14 | 99 | Y |
| simplest_design | 278 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.28 | 0.21 | -0.07 | 0.31 | 99 | Y |
| simplest_design | 279 | Q | 0 | estimator | (Intercept) | -0.24 | 0.11 | -2.22 | 0.03 | -0.45 | -0.02 | 99 | Y |
| simplest_design | 280 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.42 | 0.68 | -0.23 | 0.15 | 99 | Y |
| simplest_design | 281 | Q | 0 | estimator | (Intercept) | 0.20 | 0.11 | 1.90 | 0.06 | -0.01 | 0.41 | 99 | Y |
| simplest_design | 282 | Q | 0 | estimator | (Intercept) | 0.04 | 0.09 | 0.47 | 0.64 | -0.14 | 0.22 | 99 | Y |
| simplest_design | 283 | Q | 0 | estimator | (Intercept) | 0.18 | 0.11 | 1.68 | 0.10 | -0.03 | 0.40 | 99 | Y |
| simplest_design | 284 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.19 | 0.85 | -0.19 | 0.23 | 99 | Y |
| simplest_design | 285 | Q | 0 | estimator | (Intercept) | -0.18 | 0.09 | -2.01 | 0.05 | -0.37 | 0.00 | 99 | Y |
| simplest_design | 286 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.69 | 0.49 | -0.26 | 0.13 | 99 | Y |
| simplest_design | 287 | Q | 0 | estimator | (Intercept) | 0.09 | 0.11 | 0.81 | 0.42 | -0.13 | 0.31 | 99 | Y |
| simplest_design | 288 | Q | 0 | estimator | (Intercept) | -0.02 | 0.11 | -0.20 | 0.84 | -0.24 | 0.19 | 99 | Y |
| simplest_design | 289 | Q | 0 | estimator | (Intercept) | -0.24 | 0.09 | -2.71 | 0.01 | -0.42 | -0.06 | 99 | Y |
| simplest_design | 290 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.75 | 0.46 | -0.28 | 0.12 | 99 | Y |
| simplest_design | 291 | Q | 0 | estimator | (Intercept) | 0.12 | 0.11 | 1.05 | 0.30 | -0.11 | 0.35 | 99 | Y |
| simplest_design | 292 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.58 | 0.57 | -0.16 | 0.28 | 99 | Y |
| simplest_design | 293 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.37 | 0.17 | -0.33 | 0.06 | 99 | Y |
| simplest_design | 294 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.21 | 0.83 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 295 | Q | 0 | estimator | (Intercept) | -0.08 | 0.09 | -0.85 | 0.40 | -0.27 | 0.11 | 99 | Y |
| simplest_design | 296 | Q | 0 | estimator | (Intercept) | -0.17 | 0.11 | -1.60 | 0.11 | -0.38 | 0.04 | 99 | Y |
| simplest_design | 297 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.90 | 0.37 | -0.28 | 0.11 | 99 | Y |
| simplest_design | 298 | Q | 0 | estimator | (Intercept) | 0.14 | 0.11 | 1.23 | 0.22 | -0.08 | 0.36 | 99 | Y |
| simplest_design | 299 | Q | 0 | estimator | (Intercept) | -0.01 | 0.12 | -0.07 | 0.94 | -0.24 | 0.23 | 99 | Y |
| simplest_design | 300 | Q | 0 | estimator | (Intercept) | 0.15 | 0.12 | 1.28 | 0.20 | -0.08 | 0.39 | 99 | Y |
| simplest_design | 301 | Q | 0 | estimator | (Intercept) | -0.11 | 0.09 | -1.25 | 0.21 | -0.30 | 0.07 | 99 | Y |
| simplest_design | 302 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.20 | 0.84 | -0.20 | 0.16 | 99 | Y |
| simplest_design | 303 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.78 | 0.44 | -0.13 | 0.29 | 99 | Y |
| simplest_design | 304 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.84 | 0.40 | -0.29 | 0.12 | 99 | Y |
| simplest_design | 305 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.93 | 0.36 | -0.28 | 0.10 | 99 | Y |
| simplest_design | 306 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.98 | 0.33 | -0.28 | 0.10 | 99 | Y |
| simplest_design | 307 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.48 | 0.63 | -0.15 | 0.25 | 99 | Y |
| simplest_design | 308 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.60 | 0.55 | -0.27 | 0.15 | 99 | Y |
| simplest_design | 309 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -0.92 | 0.36 | -0.30 | 0.11 | 99 | Y |
| simplest_design | 310 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.10 | 0.92 | -0.19 | 0.21 | 99 | Y |
| simplest_design | 311 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.32 | 0.75 | -0.24 | 0.17 | 99 | Y |
| simplest_design | 312 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.24 | 0.22 | -0.08 | 0.34 | 99 | Y |
| simplest_design | 313 | Q | 0 | estimator | (Intercept) | 0.08 | 0.09 | 0.89 | 0.38 | -0.09 | 0.25 | 99 | Y |
| simplest_design | 314 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.49 | 0.62 | -0.14 | 0.23 | 99 | Y |
| simplest_design | 315 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.37 | 0.71 | -0.20 | 0.14 | 99 | Y |
| simplest_design | 316 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.30 | 0.76 | -0.23 | 0.17 | 99 | Y |
| simplest_design | 317 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | 0.01 | 0.99 | -0.17 | 0.17 | 99 | Y |
| simplest_design | 318 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -0.98 | 0.33 | -0.31 | 0.11 | 99 | Y |
| simplest_design | 319 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.55 | 0.58 | -0.22 | 0.12 | 99 | Y |
| simplest_design | 320 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.44 | 0.66 | -0.15 | 0.24 | 99 | Y |
| simplest_design | 321 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.34 | 0.74 | -0.23 | 0.16 | 99 | Y |
| simplest_design | 322 | Q | 0 | estimator | (Intercept) | 0.06 | 0.08 | 0.66 | 0.51 | -0.11 | 0.22 | 99 | Y |
| simplest_design | 323 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.40 | 0.69 | -0.25 | 0.17 | 99 | Y |
| simplest_design | 324 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -1.05 | 0.29 | -0.29 | 0.09 | 99 | Y |
| simplest_design | 325 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | -0.01 | 0.99 | -0.20 | 0.20 | 99 | Y |
| simplest_design | 326 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.60 | 0.55 | -0.22 | 0.12 | 99 | Y |
| simplest_design | 327 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.26 | 0.79 | -0.17 | 0.23 | 99 | Y |
| simplest_design | 328 | Q | 0 | estimator | (Intercept) | 0.18 | 0.09 | 1.86 | 0.07 | -0.01 | 0.36 | 99 | Y |
| simplest_design | 329 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.27 | 0.79 | -0.24 | 0.19 | 99 | Y |
| simplest_design | 330 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.03 | 0.98 | -0.20 | 0.20 | 99 | Y |
| simplest_design | 331 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.22 | 0.83 | -0.19 | 0.24 | 99 | Y |
| simplest_design | 332 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | 0.04 | 0.97 | -0.18 | 0.19 | 99 | Y |
| simplest_design | 333 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.47 | 0.64 | -0.25 | 0.15 | 99 | Y |
| simplest_design | 334 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.14 | 0.89 | -0.19 | 0.22 | 99 | Y |
| simplest_design | 335 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.16 | 0.87 | -0.19 | 0.16 | 99 | Y |
| simplest_design | 336 | Q | 0 | estimator | (Intercept) | 0.21 | 0.11 | 1.97 | 0.05 | 0.00 | 0.43 | 99 | Y |
| simplest_design | 337 | Q | 0 | estimator | (Intercept) | 0.14 | 0.10 | 1.38 | 0.17 | -0.06 | 0.35 | 99 | Y |
| simplest_design | 338 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.88 | 0.38 | -0.11 | 0.29 | 99 | Y |
| simplest_design | 339 | Q | 0 | estimator | (Intercept) | -0.17 | 0.11 | -1.52 | 0.13 | -0.39 | 0.05 | 99 | Y |
| simplest_design | 340 | Q | 0 | estimator | (Intercept) | 0.05 | 0.11 | 0.49 | 0.62 | -0.16 | 0.27 | 99 | Y |
| simplest_design | 341 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.51 | 0.61 | -0.23 | 0.13 | 99 | Y |
| simplest_design | 342 | Q | 0 | estimator | (Intercept) | -0.06 | 0.08 | -0.79 | 0.43 | -0.22 | 0.10 | 99 | Y |
| simplest_design | 343 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.34 | 0.73 | -0.15 | 0.22 | 99 | Y |
| simplest_design | 344 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.59 | 0.55 | -0.14 | 0.27 | 99 | Y |
| simplest_design | 345 | Q | 0 | estimator | (Intercept) | 0.12 | 0.09 | 1.30 | 0.20 | -0.06 | 0.31 | 99 | Y |
| simplest_design | 346 | Q | 0 | estimator | (Intercept) | -0.16 | 0.09 | -1.73 | 0.09 | -0.35 | 0.02 | 99 | Y |
| simplest_design | 347 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.79 | 0.08 | -0.38 | 0.02 | 99 | Y |
| simplest_design | 348 | Q | 0 | estimator | (Intercept) | 0.07 | 0.11 | 0.66 | 0.51 | -0.14 | 0.29 | 99 | Y |
| simplest_design | 349 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.85 | 0.40 | -0.28 | 0.11 | 99 | Y |
| simplest_design | 350 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.37 | 0.71 | -0.21 | 0.14 | 99 | Y |
| simplest_design | 351 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.61 | 0.54 | -0.26 | 0.14 | 99 | Y |
| simplest_design | 352 | Q | 0 | estimator | (Intercept) | -0.12 | 0.09 | -1.25 | 0.21 | -0.30 | 0.07 | 99 | Y |
| simplest_design | 353 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.02 | 0.98 | -0.20 | 0.20 | 99 | Y |
| simplest_design | 354 | Q | 0 | estimator | (Intercept) | 0.09 | 0.11 | 0.75 | 0.45 | -0.14 | 0.31 | 99 | Y |
| simplest_design | 355 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.29 | 0.77 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 356 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.47 | 0.64 | -0.15 | 0.24 | 99 | Y |
| simplest_design | 357 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.20 | 0.84 | -0.18 | 0.22 | 99 | Y |
| simplest_design | 358 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.73 | 0.47 | -0.11 | 0.23 | 99 | Y |
| simplest_design | 359 | Q | 0 | estimator | (Intercept) | 0.15 | 0.09 | 1.63 | 0.11 | -0.03 | 0.32 | 99 | Y |
| simplest_design | 360 | Q | 0 | estimator | (Intercept) | 0.09 | 0.11 | 0.83 | 0.41 | -0.13 | 0.31 | 99 | Y |
| simplest_design | 361 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.57 | 0.57 | -0.13 | 0.24 | 99 | Y |
| simplest_design | 362 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.11 | 0.91 | -0.21 | 0.24 | 99 | Y |
| simplest_design | 363 | Q | 0 | estimator | (Intercept) | -0.13 | 0.09 | -1.49 | 0.14 | -0.31 | 0.04 | 99 | Y |
| simplest_design | 364 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.39 | 0.70 | -0.23 | 0.15 | 99 | Y |
| simplest_design | 365 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.35 | 0.73 | -0.16 | 0.23 | 99 | Y |
| simplest_design | 366 | Q | 0 | estimator | (Intercept) | 0.07 | 0.09 | 0.74 | 0.46 | -0.12 | 0.25 | 99 | Y |
| simplest_design | 367 | Q | 0 | estimator | (Intercept) | -0.08 | 0.12 | -0.72 | 0.47 | -0.31 | 0.15 | 99 | Y |
| simplest_design | 368 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.18 | 0.86 | -0.19 | 0.23 | 99 | Y |
| simplest_design | 369 | Q | 0 | estimator | (Intercept) | 0.15 | 0.10 | 1.52 | 0.13 | -0.05 | 0.35 | 99 | Y |
| simplest_design | 370 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.61 | 0.54 | -0.25 | 0.13 | 99 | Y |
| simplest_design | 371 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.75 | 0.46 | -0.12 | 0.27 | 99 | Y |
| simplest_design | 372 | Q | 0 | estimator | (Intercept) | -0.12 | 0.11 | -1.13 | 0.26 | -0.34 | 0.09 | 99 | Y |
| simplest_design | 373 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.62 | 0.54 | -0.24 | 0.13 | 99 | Y |
| simplest_design | 374 | Q | 0 | estimator | (Intercept) | -0.08 | 0.09 | -0.85 | 0.40 | -0.27 | 0.11 | 99 | Y |
| simplest_design | 375 | Q | 0 | estimator | (Intercept) | -0.05 | 0.12 | -0.44 | 0.66 | -0.28 | 0.18 | 99 | Y |
| simplest_design | 376 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.84 | 0.40 | -0.27 | 0.11 | 99 | Y |
| simplest_design | 377 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.72 | 0.47 | -0.27 | 0.13 | 99 | Y |
| simplest_design | 378 | Q | 0 | estimator | (Intercept) | -0.01 | 0.11 | -0.11 | 0.91 | -0.23 | 0.21 | 99 | Y |
| simplest_design | 379 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.48 | 0.63 | -0.26 | 0.16 | 99 | Y |
| simplest_design | 380 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.11 | 0.91 | -0.22 | 0.19 | 99 | Y |
| simplest_design | 381 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 0.90 | 0.37 | -0.10 | 0.27 | 99 | Y |
| simplest_design | 382 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.78 | 0.08 | -0.38 | 0.02 | 99 | Y |
| simplest_design | 383 | Q | 0 | estimator | (Intercept) | 0.21 | 0.09 | 2.50 | 0.01 | 0.04 | 0.38 | 99 | Y |
| simplest_design | 384 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.37 | 0.17 | -0.34 | 0.06 | 99 | Y |
| simplest_design | 385 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.59 | 0.11 | -0.37 | 0.04 | 99 | Y |
| simplest_design | 386 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.66 | 0.51 | -0.26 | 0.13 | 99 | Y |
| simplest_design | 387 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.21 | 0.83 | -0.22 | 0.17 | 99 | Y |
| simplest_design | 388 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.58 | 0.56 | -0.27 | 0.15 | 99 | Y |
| simplest_design | 389 | Q | 0 | estimator | (Intercept) | -0.15 | 0.10 | -1.52 | 0.13 | -0.35 | 0.05 | 99 | Y |
| simplest_design | 390 | Q | 0 | estimator | (Intercept) | -0.08 | 0.11 | -0.75 | 0.46 | -0.29 | 0.13 | 99 | Y |
| simplest_design | 391 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.34 | 0.74 | -0.15 | 0.22 | 99 | Y |
| simplest_design | 392 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.78 | 0.44 | -0.12 | 0.28 | 99 | Y |
| simplest_design | 393 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.59 | 0.56 | -0.15 | 0.28 | 99 | Y |
| simplest_design | 394 | Q | 0 | estimator | (Intercept) | 0.09 | 0.08 | 1.06 | 0.29 | -0.08 | 0.26 | 99 | Y |
| simplest_design | 395 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.20 | 0.84 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 396 | Q | 0 | estimator | (Intercept) | -0.04 | 0.08 | -0.52 | 0.61 | -0.21 | 0.12 | 99 | Y |
| simplest_design | 397 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.55 | 0.58 | -0.15 | 0.27 | 99 | Y |
| simplest_design | 398 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -1.04 | 0.30 | -0.29 | 0.09 | 99 | Y |
| simplest_design | 399 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.07 | 0.95 | -0.19 | 0.21 | 99 | Y |
| simplest_design | 400 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.96 | 0.34 | -0.10 | 0.29 | 99 | Y |
| simplest_design | 401 | Q | 0 | estimator | (Intercept) | 0.12 | 0.11 | 1.16 | 0.25 | -0.09 | 0.33 | 99 | Y |
| simplest_design | 402 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.93 | 0.35 | -0.11 | 0.29 | 99 | Y |
| simplest_design | 403 | Q | 0 | estimator | (Intercept) | -0.15 | 0.12 | -1.30 | 0.20 | -0.38 | 0.08 | 99 | Y |
| simplest_design | 404 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.73 | 0.47 | -0.28 | 0.13 | 99 | Y |
| simplest_design | 405 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.20 | 0.84 | -0.17 | 0.20 | 99 | Y |
| simplest_design | 406 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.67 | 0.51 | -0.14 | 0.28 | 99 | Y |
| simplest_design | 407 | Q | 0 | estimator | (Intercept) | 0.04 | 0.09 | 0.46 | 0.65 | -0.14 | 0.22 | 99 | Y |
| simplest_design | 408 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.61 | 0.55 | -0.13 | 0.24 | 99 | Y |
| simplest_design | 409 | Q | 0 | estimator | (Intercept) | 0.01 | 0.09 | 0.15 | 0.88 | -0.17 | 0.20 | 99 | Y |
| simplest_design | 410 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.55 | 0.58 | -0.23 | 0.13 | 99 | Y |
| simplest_design | 411 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.36 | 0.72 | -0.15 | 0.22 | 99 | Y |
| simplest_design | 412 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.87 | 0.06 | -0.37 | 0.01 | 99 | Y |
| simplest_design | 413 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.23 | 0.22 | -0.08 | 0.33 | 99 | Y |
| simplest_design | 414 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.51 | 0.61 | -0.17 | 0.28 | 99 | Y |
| simplest_design | 415 | Q | 0 | estimator | (Intercept) | -0.14 | 0.09 | -1.58 | 0.12 | -0.33 | 0.04 | 99 | Y |
| simplest_design | 416 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.10 | 0.27 | -0.30 | 0.09 | 99 | Y |
| simplest_design | 417 | Q | 0 | estimator | (Intercept) | 0.17 | 0.10 | 1.74 | 0.09 | -0.02 | 0.36 | 99 | Y |
| simplest_design | 418 | Q | 0 | estimator | (Intercept) | 0.20 | 0.09 | 2.15 | 0.03 | 0.02 | 0.39 | 99 | Y |
| simplest_design | 419 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.05 | 0.96 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 420 | Q | 0 | estimator | (Intercept) | -0.09 | 0.09 | -0.92 | 0.36 | -0.27 | 0.10 | 99 | Y |
| simplest_design | 421 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.65 | 0.52 | -0.26 | 0.13 | 99 | Y |
| simplest_design | 422 | Q | 0 | estimator | (Intercept) | 0.26 | 0.10 | 2.66 | 0.01 | 0.07 | 0.45 | 99 | Y |
| simplest_design | 423 | Q | 0 | estimator | (Intercept) | -0.01 | 0.11 | -0.05 | 0.96 | -0.21 | 0.20 | 99 | Y |
| simplest_design | 424 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.34 | 0.73 | -0.23 | 0.16 | 99 | Y |
| simplest_design | 425 | Q | 0 | estimator | (Intercept) | 0.02 | 0.12 | 0.20 | 0.84 | -0.21 | 0.25 | 99 | Y |
| simplest_design | 426 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.61 | 0.54 | -0.13 | 0.25 | 99 | Y |
| simplest_design | 427 | Q | 0 | estimator | (Intercept) | 0.17 | 0.11 | 1.58 | 0.12 | -0.04 | 0.39 | 99 | Y |
| simplest_design | 428 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.35 | 0.73 | -0.22 | 0.15 | 99 | Y |
| simplest_design | 429 | Q | 0 | estimator | (Intercept) | 0.27 | 0.09 | 2.93 | 0.00 | 0.09 | 0.46 | 99 | Y |
| simplest_design | 430 | Q | 0 | estimator | (Intercept) | -0.19 | 0.09 | -1.98 | 0.05 | -0.38 | 0.00 | 99 | Y |
| simplest_design | 431 | Q | 0 | estimator | (Intercept) | -0.10 | 0.11 | -0.89 | 0.38 | -0.33 | 0.13 | 99 | Y |
| simplest_design | 432 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.38 | 0.71 | -0.16 | 0.23 | 99 | Y |
| simplest_design | 433 | Q | 0 | estimator | (Intercept) | 0.01 | 0.09 | 0.10 | 0.92 | -0.17 | 0.19 | 99 | Y |
| simplest_design | 434 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.43 | 0.67 | -0.23 | 0.15 | 99 | Y |
| simplest_design | 435 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.01 | 0.99 | -0.20 | 0.20 | 99 | Y |
| simplest_design | 436 | Q | 0 | estimator | (Intercept) | 0.05 | 0.11 | 0.49 | 0.63 | -0.16 | 0.26 | 99 | Y |
| simplest_design | 437 | Q | 0 | estimator | (Intercept) | -0.14 | 0.11 | -1.28 | 0.20 | -0.35 | 0.07 | 99 | Y |
| simplest_design | 438 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | 0.02 | 0.98 | -0.21 | 0.22 | 99 | Y |
| simplest_design | 439 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.14 | 0.89 | -0.22 | 0.19 | 99 | Y |
| simplest_design | 440 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.17 | 0.86 | -0.19 | 0.16 | 99 | Y |
| simplest_design | 441 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.86 | 0.39 | -0.11 | 0.29 | 99 | Y |
| simplest_design | 442 | Q | 0 | estimator | (Intercept) | 0.01 | 0.09 | 0.16 | 0.87 | -0.17 | 0.20 | 99 | Y |
| simplest_design | 443 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.11 | 0.91 | -0.20 | 0.22 | 99 | Y |
| simplest_design | 444 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.38 | 0.70 | -0.16 | 0.23 | 99 | Y |
| simplest_design | 445 | Q | 0 | estimator | (Intercept) | 0.19 | 0.09 | 2.12 | 0.04 | 0.01 | 0.37 | 99 | Y |
| simplest_design | 446 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.71 | 0.48 | -0.24 | 0.11 | 99 | Y |
| simplest_design | 447 | Q | 0 | estimator | (Intercept) | -0.07 | 0.11 | -0.62 | 0.53 | -0.28 | 0.15 | 99 | Y |
| simplest_design | 448 | Q | 0 | estimator | (Intercept) | 0.13 | 0.11 | 1.19 | 0.24 | -0.09 | 0.34 | 99 | Y |
| simplest_design | 449 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.58 | 0.56 | -0.14 | 0.25 | 99 | Y |
| simplest_design | 450 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.61 | 0.54 | -0.27 | 0.14 | 99 | Y |
| simplest_design | 451 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | -0.03 | 0.98 | -0.19 | 0.18 | 99 | Y |
| simplest_design | 452 | Q | 0 | estimator | (Intercept) | -0.07 | 0.11 | -0.62 | 0.53 | -0.28 | 0.14 | 99 | Y |
| simplest_design | 453 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.11 | 0.27 | -0.09 | 0.32 | 99 | Y |
| simplest_design | 454 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.32 | 0.19 | -0.32 | 0.07 | 99 | Y |
| simplest_design | 455 | Q | 0 | estimator | (Intercept) | -0.15 | 0.10 | -1.59 | 0.12 | -0.35 | 0.04 | 99 | Y |
| simplest_design | 456 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.51 | 0.61 | -0.15 | 0.25 | 99 | Y |
| simplest_design | 457 | Q | 0 | estimator | (Intercept) | -0.14 | 0.11 | -1.29 | 0.20 | -0.36 | 0.08 | 99 | Y |
| simplest_design | 458 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.36 | 0.72 | -0.23 | 0.16 | 99 | Y |
| simplest_design | 459 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.05 | 0.96 | -0.18 | 0.19 | 99 | Y |
| simplest_design | 460 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.20 | 0.84 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 461 | Q | 0 | estimator | (Intercept) | -0.08 | 0.11 | -0.77 | 0.44 | -0.29 | 0.13 | 99 | Y |
| simplest_design | 462 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.46 | 0.65 | -0.22 | 0.14 | 99 | Y |
| simplest_design | 463 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.30 | 0.77 | -0.16 | 0.22 | 99 | Y |
| simplest_design | 464 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.35 | 0.18 | -0.06 | 0.33 | 99 | Y |
| simplest_design | 465 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.37 | 0.17 | -0.33 | 0.06 | 99 | Y |
| simplest_design | 466 | Q | 0 | estimator | (Intercept) | -0.25 | 0.11 | -2.26 | 0.03 | -0.46 | -0.03 | 99 | Y |
| simplest_design | 467 | Q | 0 | estimator | (Intercept) | -0.17 | 0.10 | -1.70 | 0.09 | -0.38 | 0.03 | 99 | Y |
| simplest_design | 468 | Q | 0 | estimator | (Intercept) | 0.16 | 0.10 | 1.68 | 0.10 | -0.03 | 0.35 | 99 | Y |
| simplest_design | 469 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.20 | 0.84 | -0.21 | 0.17 | 99 | Y |
| simplest_design | 470 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.45 | 0.65 | -0.22 | 0.14 | 99 | Y |
| simplest_design | 471 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.28 | 0.78 | -0.24 | 0.18 | 99 | Y |
| simplest_design | 472 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.19 | 0.85 | -0.17 | 0.20 | 99 | Y |
| simplest_design | 473 | Q | 0 | estimator | (Intercept) | 0.07 | 0.11 | 0.67 | 0.50 | -0.14 | 0.29 | 99 | Y |
| simplest_design | 474 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.21 | 0.84 | -0.21 | 0.17 | 99 | Y |
| simplest_design | 475 | Q | 0 | estimator | (Intercept) | 0.00 | 0.12 | -0.02 | 0.98 | -0.23 | 0.23 | 99 | Y |
| simplest_design | 476 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.48 | 0.63 | -0.22 | 0.13 | 99 | Y |
| simplest_design | 477 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.48 | 0.63 | -0.16 | 0.26 | 99 | Y |
| simplest_design | 478 | Q | 0 | estimator | (Intercept) | -0.02 | 0.11 | -0.20 | 0.84 | -0.24 | 0.19 | 99 | Y |
| simplest_design | 479 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.27 | 0.79 | -0.25 | 0.19 | 99 | Y |
| simplest_design | 480 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.29 | 0.77 | -0.16 | 0.22 | 99 | Y |
| simplest_design | 481 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.61 | 0.54 | -0.24 | 0.13 | 99 | Y |
| simplest_design | 482 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.15 | 0.25 | -0.09 | 0.32 | 99 | Y |
| simplest_design | 483 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.12 | 0.91 | -0.19 | 0.22 | 99 | Y |
| simplest_design | 484 | Q | 0 | estimator | (Intercept) | -0.19 | 0.09 | -1.99 | 0.05 | -0.38 | 0.00 | 99 | Y |
| simplest_design | 485 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.21 | 0.83 | -0.20 | 0.16 | 99 | Y |
| simplest_design | 486 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.17 | 0.86 | -0.20 | 0.23 | 99 | Y |
| simplest_design | 487 | Q | 0 | estimator | (Intercept) | 0.13 | 0.08 | 1.56 | 0.12 | -0.04 | 0.30 | 99 | Y |
| simplest_design | 488 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.82 | 0.42 | -0.11 | 0.27 | 99 | Y |
| simplest_design | 489 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.29 | 0.77 | -0.24 | 0.18 | 99 | Y |
| simplest_design | 490 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.63 | 0.53 | -0.12 | 0.24 | 99 | Y |
| simplest_design | 491 | Q | 0 | estimator | (Intercept) | 0.28 | 0.08 | 3.24 | 0.00 | 0.11 | 0.44 | 99 | Y |
| simplest_design | 492 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.59 | 0.12 | -0.36 | 0.04 | 99 | Y |
| simplest_design | 493 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.18 | 0.24 | -0.32 | 0.08 | 99 | Y |
| simplest_design | 494 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | 0.01 | 0.99 | -0.22 | 0.22 | 99 | Y |
| simplest_design | 495 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.52 | 0.60 | -0.14 | 0.24 | 99 | Y |
| simplest_design | 496 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.28 | 0.78 | -0.25 | 0.19 | 99 | Y |
| simplest_design | 497 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.18 | 0.86 | -0.17 | 0.20 | 99 | Y |
| simplest_design | 498 | Q | 0 | estimator | (Intercept) | 0.08 | 0.11 | 0.73 | 0.47 | -0.13 | 0.29 | 99 | Y |
| simplest_design | 499 | Q | 0 | estimator | (Intercept) | -0.11 | 0.09 | -1.22 | 0.23 | -0.30 | 0.07 | 99 | Y |
| simplest_design | 500 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.67 | 0.51 | -0.12 | 0.24 | 99 | Y |
| simplest_design | 501 | Q | 0 | estimator | (Intercept) | 0.13 | 0.11 | 1.17 | 0.25 | -0.09 | 0.36 | 99 | Y |
| simplest_design | 502 | Q | 0 | estimator | (Intercept) | -0.12 | 0.11 | -1.15 | 0.25 | -0.33 | 0.09 | 99 | Y |
| simplest_design | 503 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.44 | 0.66 | -0.22 | 0.14 | 99 | Y |
| simplest_design | 504 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.22 | 0.82 | -0.21 | 0.17 | 99 | Y |
| simplest_design | 505 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.55 | 0.59 | -0.26 | 0.15 | 99 | Y |
| simplest_design | 506 | Q | 0 | estimator | (Intercept) | 0.03 | 0.11 | 0.30 | 0.76 | -0.18 | 0.24 | 99 | Y |
| simplest_design | 507 | Q | 0 | estimator | (Intercept) | -0.08 | 0.11 | -0.75 | 0.45 | -0.29 | 0.13 | 99 | Y |
| simplest_design | 508 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.24 | 0.81 | -0.25 | 0.20 | 99 | Y |
| simplest_design | 509 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -0.95 | 0.35 | -0.30 | 0.10 | 99 | Y |
| simplest_design | 510 | Q | 0 | estimator | (Intercept) | -0.02 | 0.11 | -0.21 | 0.84 | -0.24 | 0.19 | 99 | Y |
| simplest_design | 511 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.59 | 0.11 | -0.36 | 0.04 | 99 | Y |
| simplest_design | 512 | Q | 0 | estimator | (Intercept) | 0.15 | 0.10 | 1.48 | 0.14 | -0.05 | 0.34 | 99 | Y |
| simplest_design | 513 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.90 | 0.37 | -0.11 | 0.28 | 99 | Y |
| simplest_design | 514 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.19 | 0.85 | -0.19 | 0.23 | 99 | Y |
| simplest_design | 515 | Q | 0 | estimator | (Intercept) | 0.17 | 0.10 | 1.79 | 0.08 | -0.02 | 0.36 | 99 | Y |
| simplest_design | 516 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.43 | 0.67 | -0.25 | 0.16 | 99 | Y |
| simplest_design | 517 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.69 | 0.49 | -0.27 | 0.13 | 99 | Y |
| simplest_design | 518 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.44 | 0.66 | -0.16 | 0.25 | 99 | Y |
| simplest_design | 519 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.57 | 0.57 | -0.15 | 0.26 | 99 | Y |
| simplest_design | 520 | Q | 0 | estimator | (Intercept) | 0.15 | 0.09 | 1.69 | 0.09 | -0.03 | 0.34 | 99 | Y |
| simplest_design | 521 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.34 | 0.73 | -0.26 | 0.19 | 99 | Y |
| simplest_design | 522 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.00 | 0.32 | -0.10 | 0.31 | 99 | Y |
| simplest_design | 523 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.71 | 0.48 | -0.12 | 0.26 | 99 | Y |
| simplest_design | 524 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.43 | 0.67 | -0.26 | 0.17 | 99 | Y |
| simplest_design | 525 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.12 | 0.90 | -0.19 | 0.22 | 99 | Y |
| simplest_design | 526 | Q | 0 | estimator | (Intercept) | 0.02 | 0.12 | 0.21 | 0.84 | -0.21 | 0.25 | 99 | Y |
| simplest_design | 527 | Q | 0 | estimator | (Intercept) | 0.16 | 0.10 | 1.49 | 0.14 | -0.05 | 0.36 | 99 | Y |
| simplest_design | 528 | Q | 0 | estimator | (Intercept) | -0.22 | 0.10 | -2.18 | 0.03 | -0.42 | -0.02 | 99 | Y |
| simplest_design | 529 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.19 | 0.85 | -0.21 | 0.18 | 99 | Y |
| simplest_design | 530 | Q | 0 | estimator | (Intercept) | 0.08 | 0.11 | 0.76 | 0.45 | -0.13 | 0.30 | 99 | Y |
| simplest_design | 531 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.38 | 0.71 | -0.26 | 0.17 | 99 | Y |
| simplest_design | 532 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.12 | 0.90 | -0.21 | 0.19 | 99 | Y |
| simplest_design | 533 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.22 | 0.82 | -0.19 | 0.24 | 99 | Y |
| simplest_design | 534 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.36 | 0.72 | -0.27 | 0.19 | 99 | Y |
| simplest_design | 535 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.36 | 0.18 | -0.33 | 0.06 | 99 | Y |
| simplest_design | 536 | Q | 0 | estimator | (Intercept) | -0.16 | 0.09 | -1.67 | 0.10 | -0.34 | 0.03 | 99 | Y |
| simplest_design | 537 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.82 | 0.42 | -0.28 | 0.12 | 99 | Y |
| simplest_design | 538 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.63 | 0.53 | -0.24 | 0.12 | 99 | Y |
| simplest_design | 539 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.23 | 0.82 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 540 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.20 | 0.84 | -0.20 | 0.24 | 99 | Y |
| simplest_design | 541 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.03 | 0.98 | -0.19 | 0.20 | 99 | Y |
| simplest_design | 542 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.55 | 0.58 | -0.25 | 0.14 | 99 | Y |
| simplest_design | 543 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.15 | 0.88 | -0.22 | 0.19 | 99 | Y |
| simplest_design | 544 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.35 | 0.72 | -0.22 | 0.16 | 99 | Y |
| simplest_design | 545 | Q | 0 | estimator | (Intercept) | -0.13 | 0.09 | -1.54 | 0.13 | -0.30 | 0.04 | 99 | Y |
| simplest_design | 546 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.04 | 0.97 | -0.20 | 0.21 | 99 | Y |
| simplest_design | 547 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.26 | 0.80 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 548 | Q | 0 | estimator | (Intercept) | 0.07 | 0.11 | 0.69 | 0.49 | -0.14 | 0.29 | 99 | Y |
| simplest_design | 549 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.64 | 0.53 | -0.26 | 0.13 | 99 | Y |
| simplest_design | 550 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.62 | 0.54 | -0.27 | 0.14 | 99 | Y |
| simplest_design | 551 | Q | 0 | estimator | (Intercept) | -0.07 | 0.09 | -0.75 | 0.45 | -0.25 | 0.11 | 99 | Y |
| simplest_design | 552 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.13 | 0.89 | -0.19 | 0.21 | 99 | Y |
| simplest_design | 553 | Q | 0 | estimator | (Intercept) | -0.02 | 0.08 | -0.20 | 0.84 | -0.18 | 0.15 | 99 | Y |
| simplest_design | 554 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.72 | 0.47 | -0.28 | 0.13 | 99 | Y |
| simplest_design | 555 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.78 | 0.44 | -0.12 | 0.27 | 99 | Y |
| simplest_design | 556 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | -0.02 | 0.98 | -0.18 | 0.18 | 99 | Y |
| simplest_design | 557 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.45 | 0.65 | -0.23 | 0.15 | 99 | Y |
| simplest_design | 558 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.21 | 0.83 | -0.17 | 0.21 | 99 | Y |
| simplest_design | 559 | Q | 0 | estimator | (Intercept) | 0.16 | 0.09 | 1.76 | 0.08 | -0.02 | 0.35 | 99 | Y |
| simplest_design | 560 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.72 | 0.47 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 561 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.81 | 0.42 | -0.12 | 0.28 | 99 | Y |
| simplest_design | 562 | Q | 0 | estimator | (Intercept) | -0.23 | 0.10 | -2.15 | 0.03 | -0.43 | -0.02 | 99 | Y |
| simplest_design | 563 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.13 | 0.90 | -0.20 | 0.23 | 99 | Y |
| simplest_design | 564 | Q | 0 | estimator | (Intercept) | -0.07 | 0.11 | -0.63 | 0.53 | -0.29 | 0.15 | 99 | Y |
| simplest_design | 565 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.20 | 0.84 | -0.19 | 0.16 | 99 | Y |
| simplest_design | 566 | Q | 0 | estimator | (Intercept) | -0.13 | 0.11 | -1.20 | 0.23 | -0.34 | 0.08 | 99 | Y |
| simplest_design | 567 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.86 | 0.07 | -0.37 | 0.01 | 99 | Y |
| simplest_design | 568 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.59 | 0.12 | -0.37 | 0.04 | 99 | Y |
| simplest_design | 569 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.59 | 0.56 | -0.14 | 0.26 | 99 | Y |
| simplest_design | 570 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 1.01 | 0.31 | -0.09 | 0.28 | 99 | Y |
| simplest_design | 571 | Q | 0 | estimator | (Intercept) | 0.21 | 0.10 | 2.05 | 0.04 | 0.01 | 0.41 | 99 | Y |
| simplest_design | 572 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.69 | 0.49 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 573 | Q | 0 | estimator | (Intercept) | 0.18 | 0.09 | 1.87 | 0.06 | -0.01 | 0.36 | 99 | Y |
| simplest_design | 574 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.25 | 0.22 | -0.07 | 0.31 | 99 | Y |
| simplest_design | 575 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 1.03 | 0.30 | -0.09 | 0.27 | 99 | Y |
| simplest_design | 576 | Q | 0 | estimator | (Intercept) | 0.04 | 0.09 | 0.45 | 0.66 | -0.14 | 0.22 | 99 | Y |
| simplest_design | 577 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.65 | 0.52 | -0.14 | 0.27 | 99 | Y |
| simplest_design | 578 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.18 | 0.24 | -0.08 | 0.31 | 99 | Y |
| simplest_design | 579 | Q | 0 | estimator | (Intercept) | -0.17 | 0.11 | -1.64 | 0.10 | -0.39 | 0.04 | 99 | Y |
| simplest_design | 580 | Q | 0 | estimator | (Intercept) | -0.16 | 0.09 | -1.71 | 0.09 | -0.35 | 0.03 | 99 | Y |
| simplest_design | 581 | Q | 0 | estimator | (Intercept) | 0.13 | 0.12 | 1.02 | 0.31 | -0.12 | 0.37 | 99 | Y |
| simplest_design | 582 | Q | 0 | estimator | (Intercept) | 0.11 | 0.11 | 1.04 | 0.30 | -0.10 | 0.32 | 99 | Y |
| simplest_design | 583 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.05 | 0.96 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 584 | Q | 0 | estimator | (Intercept) | 0.07 | 0.11 | 0.67 | 0.51 | -0.14 | 0.29 | 99 | Y |
| simplest_design | 585 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.24 | 0.81 | -0.22 | 0.17 | 99 | Y |
| simplest_design | 586 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.70 | 0.48 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 587 | Q | 0 | estimator | (Intercept) | 0.01 | 0.12 | 0.05 | 0.96 | -0.23 | 0.25 | 99 | Y |
| simplest_design | 588 | Q | 0 | estimator | (Intercept) | -0.25 | 0.11 | -2.32 | 0.02 | -0.46 | -0.04 | 99 | Y |
| simplest_design | 589 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.34 | 0.18 | -0.33 | 0.06 | 99 | Y |
| simplest_design | 590 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.46 | 0.64 | -0.23 | 0.14 | 99 | Y |
| simplest_design | 591 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.28 | 0.78 | -0.22 | 0.17 | 99 | Y |
| simplest_design | 592 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.25 | 0.80 | -0.16 | 0.21 | 99 | Y |
| simplest_design | 593 | Q | 0 | estimator | (Intercept) | 0.23 | 0.10 | 2.21 | 0.03 | 0.02 | 0.43 | 99 | Y |
| simplest_design | 594 | Q | 0 | estimator | (Intercept) | 0.05 | 0.11 | 0.40 | 0.69 | -0.18 | 0.27 | 99 | Y |
| simplest_design | 595 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.30 | 0.77 | -0.17 | 0.23 | 99 | Y |
| simplest_design | 596 | Q | 0 | estimator | (Intercept) | 0.10 | 0.09 | 1.10 | 0.27 | -0.08 | 0.29 | 99 | Y |
| simplest_design | 597 | Q | 0 | estimator | (Intercept) | -0.11 | 0.09 | -1.18 | 0.24 | -0.30 | 0.08 | 99 | Y |
| simplest_design | 598 | Q | 0 | estimator | (Intercept) | 0.08 | 0.11 | 0.77 | 0.44 | -0.13 | 0.30 | 99 | Y |
| simplest_design | 599 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.18 | 0.86 | -0.20 | 0.16 | 99 | Y |
| simplest_design | 600 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.76 | 0.45 | -0.12 | 0.27 | 99 | Y |
| simplest_design | 601 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.36 | 0.72 | -0.21 | 0.14 | 99 | Y |
| simplest_design | 602 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.44 | 0.15 | -0.34 | 0.05 | 99 | Y |
| simplest_design | 603 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.33 | 0.74 | -0.23 | 0.17 | 99 | Y |
| simplest_design | 604 | Q | 0 | estimator | (Intercept) | -0.10 | 0.09 | -1.05 | 0.30 | -0.28 | 0.09 | 99 | Y |
| simplest_design | 605 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.04 | 0.97 | -0.20 | 0.21 | 99 | Y |
| simplest_design | 606 | Q | 0 | estimator | (Intercept) | 0.08 | 0.08 | 0.93 | 0.36 | -0.09 | 0.24 | 99 | Y |
| simplest_design | 607 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.70 | 0.48 | -0.26 | 0.13 | 99 | Y |
| simplest_design | 608 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.73 | 0.47 | -0.13 | 0.28 | 99 | Y |
| simplest_design | 609 | Q | 0 | estimator | (Intercept) | -0.07 | 0.11 | -0.65 | 0.51 | -0.28 | 0.14 | 99 | Y |
| simplest_design | 610 | Q | 0 | estimator | (Intercept) | 0.19 | 0.10 | 1.91 | 0.06 | -0.01 | 0.40 | 99 | Y |
| simplest_design | 611 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.11 | 0.27 | -0.32 | 0.09 | 99 | Y |
| simplest_design | 612 | Q | 0 | estimator | (Intercept) | -0.22 | 0.11 | -2.04 | 0.04 | -0.43 | -0.01 | 99 | Y |
| simplest_design | 613 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.12 | 0.90 | -0.20 | 0.23 | 99 | Y |
| simplest_design | 614 | Q | 0 | estimator | (Intercept) | 0.01 | 0.09 | 0.08 | 0.94 | -0.18 | 0.19 | 99 | Y |
| simplest_design | 615 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.88 | 0.38 | -0.11 | 0.27 | 99 | Y |
| simplest_design | 616 | Q | 0 | estimator | (Intercept) | -0.13 | 0.12 | -1.11 | 0.27 | -0.37 | 0.10 | 99 | Y |
| simplest_design | 617 | Q | 0 | estimator | (Intercept) | -0.08 | 0.11 | -0.74 | 0.46 | -0.30 | 0.14 | 99 | Y |
| simplest_design | 618 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.18 | 0.85 | -0.18 | 0.22 | 99 | Y |
| simplest_design | 619 | Q | 0 | estimator | (Intercept) | 0.03 | 0.11 | 0.31 | 0.76 | -0.18 | 0.24 | 99 | Y |
| simplest_design | 620 | Q | 0 | estimator | (Intercept) | 0.10 | 0.09 | 1.11 | 0.27 | -0.08 | 0.28 | 99 | Y |
| simplest_design | 621 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.13 | 0.26 | -0.30 | 0.08 | 99 | Y |
| simplest_design | 622 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.32 | 0.75 | -0.17 | 0.23 | 99 | Y |
| simplest_design | 623 | Q | 0 | estimator | (Intercept) | -0.17 | 0.09 | -1.83 | 0.07 | -0.35 | 0.01 | 99 | Y |
| simplest_design | 624 | Q | 0 | estimator | (Intercept) | 0.17 | 0.10 | 1.68 | 0.10 | -0.03 | 0.36 | 99 | Y |
| simplest_design | 625 | Q | 0 | estimator | (Intercept) | 0.16 | 0.09 | 1.75 | 0.08 | -0.02 | 0.35 | 99 | Y |
| simplest_design | 626 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.17 | 0.87 | -0.19 | 0.22 | 99 | Y |
| simplest_design | 627 | Q | 0 | estimator | (Intercept) | -0.15 | 0.11 | -1.42 | 0.16 | -0.37 | 0.06 | 99 | Y |
| simplest_design | 628 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.83 | 0.41 | -0.27 | 0.11 | 99 | Y |
| simplest_design | 629 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.08 | 0.93 | -0.18 | 0.20 | 99 | Y |
| simplest_design | 630 | Q | 0 | estimator | (Intercept) | -0.15 | 0.10 | -1.45 | 0.15 | -0.36 | 0.06 | 99 | Y |
| simplest_design | 631 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.11 | 0.91 | -0.19 | 0.17 | 99 | Y |
| simplest_design | 632 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.13 | 0.90 | -0.18 | 0.21 | 99 | Y |
| simplest_design | 633 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.16 | 0.25 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 634 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.59 | 0.56 | -0.25 | 0.13 | 99 | Y |
| simplest_design | 635 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.62 | 0.54 | -0.13 | 0.26 | 99 | Y |
| simplest_design | 636 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.46 | 0.65 | -0.22 | 0.14 | 99 | Y |
| simplest_design | 637 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.47 | 0.64 | -0.26 | 0.16 | 99 | Y |
| simplest_design | 638 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.73 | 0.47 | -0.13 | 0.28 | 99 | Y |
| simplest_design | 639 | Q | 0 | estimator | (Intercept) | 0.07 | 0.11 | 0.67 | 0.51 | -0.15 | 0.30 | 99 | Y |
| simplest_design | 640 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.34 | 0.74 | -0.23 | 0.16 | 99 | Y |
| simplest_design | 641 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 0.96 | 0.34 | -0.10 | 0.30 | 99 | Y |
| simplest_design | 642 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.11 | 0.92 | -0.19 | 0.17 | 99 | Y |
| simplest_design | 643 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.90 | 0.37 | -0.30 | 0.11 | 99 | Y |
| simplest_design | 644 | Q | 0 | estimator | (Intercept) | -0.04 | 0.08 | -0.52 | 0.60 | -0.21 | 0.12 | 99 | Y |
| simplest_design | 645 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | 0.04 | 0.97 | -0.21 | 0.22 | 99 | Y |
| simplest_design | 646 | Q | 0 | estimator | (Intercept) | -0.17 | 0.11 | -1.53 | 0.13 | -0.40 | 0.05 | 99 | Y |
| simplest_design | 647 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.18 | 0.86 | -0.22 | 0.19 | 99 | Y |
| simplest_design | 648 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.28 | 0.78 | -0.24 | 0.18 | 99 | Y |
| simplest_design | 649 | Q | 0 | estimator | (Intercept) | 0.10 | 0.11 | 0.89 | 0.38 | -0.12 | 0.31 | 99 | Y |
| simplest_design | 650 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.24 | 0.81 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 651 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.52 | 0.60 | -0.27 | 0.16 | 99 | Y |
| simplest_design | 652 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.66 | 0.51 | -0.13 | 0.26 | 99 | Y |
| simplest_design | 653 | Q | 0 | estimator | (Intercept) | -0.14 | 0.09 | -1.57 | 0.12 | -0.32 | 0.04 | 99 | Y |
| simplest_design | 654 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.11 | 0.91 | -0.21 | 0.19 | 99 | Y |
| simplest_design | 655 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.05 | 0.29 | -0.09 | 0.29 | 99 | Y |
| simplest_design | 656 | Q | 0 | estimator | (Intercept) | -0.23 | 0.09 | -2.50 | 0.01 | -0.41 | -0.05 | 99 | Y |
| simplest_design | 657 | Q | 0 | estimator | (Intercept) | -0.19 | 0.12 | -1.63 | 0.11 | -0.42 | 0.04 | 99 | Y |
| simplest_design | 658 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.05 | 0.96 | -0.19 | 0.19 | 99 | Y |
| simplest_design | 659 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.18 | 0.24 | -0.08 | 0.30 | 99 | Y |
| simplest_design | 660 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | -0.01 | 0.99 | -0.18 | 0.17 | 99 | Y |
| simplest_design | 661 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.09 | 0.28 | -0.09 | 0.32 | 99 | Y |
| simplest_design | 662 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.30 | 0.77 | -0.25 | 0.18 | 99 | Y |
| simplest_design | 663 | Q | 0 | estimator | (Intercept) | 0.15 | 0.10 | 1.48 | 0.14 | -0.05 | 0.35 | 99 | Y |
| simplest_design | 664 | Q | 0 | estimator | (Intercept) | 0.07 | 0.11 | 0.64 | 0.52 | -0.15 | 0.29 | 99 | Y |
| simplest_design | 665 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.13 | 0.26 | -0.08 | 0.30 | 99 | Y |
| simplest_design | 666 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.08 | 0.94 | -0.22 | 0.20 | 99 | Y |
| simplest_design | 667 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.10 | 0.27 | -0.09 | 0.32 | 99 | Y |
| simplest_design | 668 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.32 | 0.75 | -0.25 | 0.18 | 99 | Y |
| simplest_design | 669 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.29 | 0.77 | -0.25 | 0.19 | 99 | Y |
| simplest_design | 670 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.72 | 0.48 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 671 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.36 | 0.18 | -0.33 | 0.06 | 99 | Y |
| simplest_design | 672 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.60 | 0.55 | -0.24 | 0.13 | 99 | Y |
| simplest_design | 673 | Q | 0 | estimator | (Intercept) | 0.07 | 0.09 | 0.71 | 0.48 | -0.12 | 0.25 | 99 | Y |
| simplest_design | 674 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.28 | 0.78 | -0.24 | 0.18 | 99 | Y |
| simplest_design | 675 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.47 | 0.64 | -0.27 | 0.16 | 99 | Y |
| simplest_design | 676 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.56 | 0.58 | -0.24 | 0.14 | 99 | Y |
| simplest_design | 677 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.12 | 0.27 | -0.31 | 0.09 | 99 | Y |
| simplest_design | 678 | Q | 0 | estimator | (Intercept) | 0.14 | 0.11 | 1.25 | 0.22 | -0.08 | 0.36 | 99 | Y |
| simplest_design | 679 | Q | 0 | estimator | (Intercept) | 0.12 | 0.09 | 1.26 | 0.21 | -0.07 | 0.31 | 99 | Y |
| simplest_design | 680 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.67 | 0.51 | -0.13 | 0.25 | 99 | Y |
| simplest_design | 681 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.53 | 0.59 | -0.26 | 0.15 | 99 | Y |
| simplest_design | 682 | Q | 0 | estimator | (Intercept) | 0.23 | 0.10 | 2.31 | 0.02 | 0.03 | 0.43 | 99 | Y |
| simplest_design | 683 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.72 | 0.47 | -0.28 | 0.13 | 99 | Y |
| simplest_design | 684 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.94 | 0.35 | -0.29 | 0.10 | 99 | Y |
| simplest_design | 685 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.13 | 0.90 | -0.19 | 0.22 | 99 | Y |
| simplest_design | 686 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.83 | 0.41 | -0.12 | 0.29 | 99 | Y |
| simplest_design | 687 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.47 | 0.64 | -0.25 | 0.16 | 99 | Y |
| simplest_design | 688 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.05 | 0.96 | -0.21 | 0.22 | 99 | Y |
| simplest_design | 689 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.28 | 0.78 | -0.16 | 0.21 | 99 | Y |
| simplest_design | 690 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.67 | 0.50 | -0.13 | 0.26 | 99 | Y |
| simplest_design | 691 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -0.94 | 0.35 | -0.31 | 0.11 | 99 | Y |
| simplest_design | 692 | Q | 0 | estimator | (Intercept) | -0.15 | 0.10 | -1.55 | 0.12 | -0.34 | 0.04 | 99 | Y |
| simplest_design | 693 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | -0.01 | 0.99 | -0.19 | 0.19 | 99 | Y |
| simplest_design | 694 | Q | 0 | estimator | (Intercept) | -0.11 | 0.11 | -1.07 | 0.29 | -0.32 | 0.10 | 99 | Y |
| simplest_design | 695 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.34 | 0.73 | -0.23 | 0.16 | 99 | Y |
| simplest_design | 696 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.36 | 0.72 | -0.22 | 0.15 | 99 | Y |
| simplest_design | 697 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 0.99 | 0.32 | -0.09 | 0.28 | 99 | Y |
| simplest_design | 698 | Q | 0 | estimator | (Intercept) | 0.00 | 0.12 | -0.04 | 0.97 | -0.23 | 0.22 | 99 | Y |
| simplest_design | 699 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.11 | 0.91 | -0.21 | 0.19 | 99 | Y |
| simplest_design | 700 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.42 | 0.68 | -0.15 | 0.23 | 99 | Y |
| simplest_design | 701 | Q | 0 | estimator | (Intercept) | -0.33 | 0.10 | -3.20 | 0.00 | -0.53 | -0.12 | 99 | Y |
| simplest_design | 702 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.74 | 0.46 | -0.27 | 0.12 | 99 | Y |
| simplest_design | 703 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.89 | 0.37 | -0.30 | 0.11 | 99 | Y |
| simplest_design | 704 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.48 | 0.63 | -0.25 | 0.15 | 99 | Y |
| simplest_design | 705 | Q | 0 | estimator | (Intercept) | 0.22 | 0.09 | 2.41 | 0.02 | 0.04 | 0.40 | 99 | Y |
| simplest_design | 706 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.21 | 0.83 | -0.17 | 0.21 | 99 | Y |
| simplest_design | 707 | Q | 0 | estimator | (Intercept) | 0.23 | 0.09 | 2.49 | 0.01 | 0.05 | 0.40 | 99 | Y |
| simplest_design | 708 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.20 | 0.23 | -0.08 | 0.31 | 99 | Y |
| simplest_design | 709 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.15 | 0.88 | -0.21 | 0.18 | 99 | Y |
| simplest_design | 710 | Q | 0 | estimator | (Intercept) | -0.14 | 0.09 | -1.55 | 0.13 | -0.33 | 0.04 | 99 | Y |
| simplest_design | 711 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.45 | 0.65 | -0.21 | 0.13 | 99 | Y |
| simplest_design | 712 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | -0.04 | 0.96 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 713 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.27 | 0.21 | -0.07 | 0.33 | 99 | Y |
| simplest_design | 714 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.53 | 0.60 | -0.23 | 0.14 | 99 | Y |
| simplest_design | 715 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | -0.04 | 0.97 | -0.23 | 0.22 | 99 | Y |
| simplest_design | 716 | Q | 0 | estimator | (Intercept) | 0.14 | 0.11 | 1.30 | 0.20 | -0.07 | 0.36 | 99 | Y |
| simplest_design | 717 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.39 | 0.70 | -0.14 | 0.21 | 99 | Y |
| simplest_design | 718 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.61 | 0.54 | -0.13 | 0.25 | 99 | Y |
| simplest_design | 719 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.46 | 0.65 | -0.24 | 0.15 | 99 | Y |
| simplest_design | 720 | Q | 0 | estimator | (Intercept) | 0.09 | 0.09 | 0.97 | 0.34 | -0.09 | 0.27 | 99 | Y |
| simplest_design | 721 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.20 | 0.84 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 722 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.71 | 0.48 | -0.12 | 0.26 | 99 | Y |
| simplest_design | 723 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.16 | 0.25 | -0.32 | 0.08 | 99 | Y |
| simplest_design | 724 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.47 | 0.64 | -0.21 | 0.13 | 99 | Y |
| simplest_design | 725 | Q | 0 | estimator | (Intercept) | 0.04 | 0.11 | 0.35 | 0.73 | -0.17 | 0.25 | 99 | Y |
| simplest_design | 726 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.89 | 0.37 | -0.30 | 0.11 | 99 | Y |
| simplest_design | 727 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.68 | 0.50 | -0.13 | 0.26 | 99 | Y |
| simplest_design | 728 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.09 | 0.28 | -0.09 | 0.32 | 99 | Y |
| simplest_design | 729 | Q | 0 | estimator | (Intercept) | -0.03 | 0.09 | -0.37 | 0.71 | -0.22 | 0.15 | 99 | Y |
| simplest_design | 730 | Q | 0 | estimator | (Intercept) | 0.08 | 0.11 | 0.77 | 0.44 | -0.13 | 0.30 | 99 | Y |
| simplest_design | 731 | Q | 0 | estimator | (Intercept) | -0.12 | 0.09 | -1.37 | 0.17 | -0.30 | 0.05 | 99 | Y |
| simplest_design | 732 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.27 | 0.79 | -0.16 | 0.21 | 99 | Y |
| simplest_design | 733 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.64 | 0.10 | -0.36 | 0.03 | 99 | Y |
| simplest_design | 734 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.08 | 0.94 | -0.19 | 0.18 | 99 | Y |
| simplest_design | 735 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.09 | 0.93 | -0.18 | 0.20 | 99 | Y |
| simplest_design | 736 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.32 | 0.75 | -0.16 | 0.23 | 99 | Y |
| simplest_design | 737 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.49 | 0.63 | -0.24 | 0.14 | 99 | Y |
| simplest_design | 738 | Q | 0 | estimator | (Intercept) | -0.17 | 0.10 | -1.76 | 0.08 | -0.36 | 0.02 | 99 | Y |
| simplest_design | 739 | Q | 0 | estimator | (Intercept) | -0.33 | 0.10 | -3.35 | 0.00 | -0.52 | -0.13 | 99 | Y |
| simplest_design | 740 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.81 | 0.42 | -0.28 | 0.12 | 99 | Y |
| simplest_design | 741 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | 0.01 | 0.99 | -0.22 | 0.22 | 99 | Y |
| simplest_design | 742 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.06 | 0.96 | -0.18 | 0.19 | 99 | Y |
| simplest_design | 743 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.66 | 0.51 | -0.13 | 0.26 | 99 | Y |
| simplest_design | 744 | Q | 0 | estimator | (Intercept) | 0.00 | 0.12 | -0.04 | 0.97 | -0.23 | 0.22 | 99 | Y |
| simplest_design | 745 | Q | 0 | estimator | (Intercept) | 0.12 | 0.09 | 1.27 | 0.21 | -0.07 | 0.30 | 99 | Y |
| simplest_design | 746 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.03 | 0.98 | -0.19 | 0.20 | 99 | Y |
| simplest_design | 747 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.18 | 0.85 | -0.18 | 0.22 | 99 | Y |
| simplest_design | 748 | Q | 0 | estimator | (Intercept) | -0.17 | 0.09 | -1.83 | 0.07 | -0.35 | 0.01 | 99 | Y |
| simplest_design | 749 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.83 | 0.07 | -0.38 | 0.01 | 99 | Y |
| simplest_design | 750 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.72 | 0.47 | -0.27 | 0.13 | 99 | Y |
| simplest_design | 751 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 0.99 | 0.32 | -0.10 | 0.29 | 99 | Y |
| simplest_design | 752 | Q | 0 | estimator | (Intercept) | 0.14 | 0.11 | 1.32 | 0.19 | -0.07 | 0.36 | 99 | Y |
| simplest_design | 753 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.23 | 0.22 | -0.31 | 0.07 | 99 | Y |
| simplest_design | 754 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.18 | 0.86 | -0.23 | 0.19 | 99 | Y |
| simplest_design | 755 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.11 | 0.92 | -0.18 | 0.20 | 99 | Y |
| simplest_design | 756 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | -0.02 | 0.99 | -0.21 | 0.21 | 99 | Y |
| simplest_design | 757 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.04 | 0.97 | -0.20 | 0.21 | 99 | Y |
| simplest_design | 758 | Q | 0 | estimator | (Intercept) | -0.23 | 0.10 | -2.30 | 0.02 | -0.42 | -0.03 | 99 | Y |
| simplest_design | 759 | Q | 0 | estimator | (Intercept) | 0.18 | 0.10 | 1.78 | 0.08 | -0.02 | 0.38 | 99 | Y |
| simplest_design | 760 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.09 | 0.93 | -0.20 | 0.22 | 99 | Y |
| simplest_design | 761 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.17 | 0.24 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 762 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.30 | 0.20 | -0.07 | 0.33 | 99 | Y |
| simplest_design | 763 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.70 | 0.49 | -0.28 | 0.14 | 99 | Y |
| simplest_design | 764 | Q | 0 | estimator | (Intercept) | -0.07 | 0.09 | -0.76 | 0.45 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 765 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | 0.03 | 0.98 | -0.17 | 0.18 | 99 | Y |
| simplest_design | 766 | Q | 0 | estimator | (Intercept) | -0.13 | 0.10 | -1.21 | 0.23 | -0.33 | 0.08 | 99 | Y |
| simplest_design | 767 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.21 | 0.83 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 768 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.37 | 0.72 | -0.25 | 0.17 | 99 | Y |
| simplest_design | 769 | Q | 0 | estimator | (Intercept) | -0.24 | 0.10 | -2.45 | 0.02 | -0.43 | -0.05 | 99 | Y |
| simplest_design | 770 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.50 | 0.62 | -0.28 | 0.16 | 99 | Y |
| simplest_design | 771 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.31 | 0.19 | -0.07 | 0.33 | 99 | Y |
| simplest_design | 772 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.43 | 0.67 | -0.27 | 0.17 | 99 | Y |
| simplest_design | 773 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.07 | 0.94 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 774 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.60 | 0.55 | -0.28 | 0.15 | 99 | Y |
| simplest_design | 775 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | 0.02 | 0.98 | -0.18 | 0.18 | 99 | Y |
| simplest_design | 776 | Q | 0 | estimator | (Intercept) | 0.03 | 0.11 | 0.28 | 0.78 | -0.18 | 0.24 | 99 | Y |
| simplest_design | 777 | Q | 0 | estimator | (Intercept) | 0.03 | 0.11 | 0.27 | 0.79 | -0.18 | 0.24 | 99 | Y |
| simplest_design | 778 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.37 | 0.71 | -0.16 | 0.23 | 99 | Y |
| simplest_design | 779 | Q | 0 | estimator | (Intercept) | 0.05 | 0.11 | 0.49 | 0.63 | -0.17 | 0.28 | 99 | Y |
| simplest_design | 780 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.61 | 0.54 | -0.13 | 0.24 | 99 | Y |
| simplest_design | 781 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.83 | 0.41 | -0.28 | 0.11 | 99 | Y |
| simplest_design | 782 | Q | 0 | estimator | (Intercept) | 0.04 | 0.09 | 0.44 | 0.66 | -0.13 | 0.21 | 99 | Y |
| simplest_design | 783 | Q | 0 | estimator | (Intercept) | 0.12 | 0.11 | 1.17 | 0.25 | -0.09 | 0.33 | 99 | Y |
| simplest_design | 784 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.01 | 0.99 | -0.20 | 0.20 | 99 | Y |
| simplest_design | 785 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.02 | 0.31 | -0.09 | 0.29 | 99 | Y |
| simplest_design | 786 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.78 | 0.44 | -0.27 | 0.12 | 99 | Y |
| simplest_design | 787 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.22 | 0.82 | -0.20 | 0.25 | 99 | Y |
| simplest_design | 788 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.82 | 0.41 | -0.12 | 0.28 | 99 | Y |
| simplest_design | 789 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.13 | 0.26 | -0.32 | 0.09 | 99 | Y |
| simplest_design | 790 | Q | 0 | estimator | (Intercept) | -0.31 | 0.10 | -3.03 | 0.00 | -0.51 | -0.11 | 99 | Y |
| simplest_design | 791 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.43 | 0.67 | -0.22 | 0.14 | 99 | Y |
| simplest_design | 792 | Q | 0 | estimator | (Intercept) | 0.20 | 0.09 | 2.14 | 0.03 | 0.02 | 0.39 | 99 | Y |
| simplest_design | 793 | Q | 0 | estimator | (Intercept) | 0.16 | 0.09 | 1.83 | 0.07 | -0.01 | 0.34 | 99 | Y |
| simplest_design | 794 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.11 | 0.91 | -0.20 | 0.18 | 99 | Y |
| simplest_design | 795 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.20 | 0.84 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 796 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.44 | 0.66 | -0.24 | 0.16 | 99 | Y |
| simplest_design | 797 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.06 | 0.95 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 798 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 0.93 | 0.36 | -0.11 | 0.30 | 99 | Y |
| simplest_design | 799 | Q | 0 | estimator | (Intercept) | 0.19 | 0.10 | 1.94 | 0.06 | 0.00 | 0.38 | 99 | Y |
| simplest_design | 800 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.13 | 0.89 | -0.21 | 0.18 | 99 | Y |
| simplest_design | 801 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.96 | 0.34 | -0.10 | 0.28 | 99 | Y |
| simplest_design | 802 | Q | 0 | estimator | (Intercept) | 0.01 | 0.11 | 0.07 | 0.95 | -0.20 | 0.22 | 99 | Y |
| simplest_design | 803 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.75 | 0.46 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 804 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.94 | 0.35 | -0.10 | 0.29 | 99 | Y |
| simplest_design | 805 | Q | 0 | estimator | (Intercept) | -0.09 | 0.11 | -0.86 | 0.39 | -0.30 | 0.12 | 99 | Y |
| simplest_design | 806 | Q | 0 | estimator | (Intercept) | 0.05 | 0.12 | 0.46 | 0.64 | -0.18 | 0.28 | 99 | Y |
| simplest_design | 807 | Q | 0 | estimator | (Intercept) | -0.22 | 0.11 | -2.00 | 0.05 | -0.44 | 0.00 | 99 | Y |
| simplest_design | 808 | Q | 0 | estimator | (Intercept) | -0.28 | 0.09 | -3.21 | 0.00 | -0.45 | -0.11 | 99 | Y |
| simplest_design | 809 | Q | 0 | estimator | (Intercept) | -0.16 | 0.11 | -1.49 | 0.14 | -0.37 | 0.05 | 99 | Y |
| simplest_design | 810 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.15 | 0.88 | -0.20 | 0.24 | 99 | Y |
| simplest_design | 811 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | -0.05 | 0.96 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 812 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.47 | 0.64 | -0.16 | 0.25 | 99 | Y |
| simplest_design | 813 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.46 | 0.65 | -0.26 | 0.16 | 99 | Y |
| simplest_design | 814 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.98 | 0.33 | -0.10 | 0.29 | 99 | Y |
| simplest_design | 815 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.89 | 0.38 | -0.11 | 0.28 | 99 | Y |
| simplest_design | 816 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | -0.02 | 0.99 | -0.23 | 0.23 | 99 | Y |
| simplest_design | 817 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.25 | 0.80 | -0.23 | 0.18 | 99 | Y |
| simplest_design | 818 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.03 | 0.31 | -0.09 | 0.30 | 99 | Y |
| simplest_design | 819 | Q | 0 | estimator | (Intercept) | 0.20 | 0.09 | 2.26 | 0.03 | 0.02 | 0.38 | 99 | Y |
| simplest_design | 820 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.15 | 0.88 | -0.20 | 0.17 | 99 | Y |
| simplest_design | 821 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.74 | 0.46 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 822 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.42 | 0.68 | -0.26 | 0.17 | 99 | Y |
| simplest_design | 823 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.77 | 0.44 | -0.12 | 0.26 | 99 | Y |
| simplest_design | 824 | Q | 0 | estimator | (Intercept) | 0.16 | 0.09 | 1.71 | 0.09 | -0.03 | 0.34 | 99 | Y |
| simplest_design | 825 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.43 | 0.67 | -0.24 | 0.16 | 99 | Y |
| simplest_design | 826 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.36 | 0.72 | -0.24 | 0.17 | 99 | Y |
| simplest_design | 827 | Q | 0 | estimator | (Intercept) | 0.09 | 0.11 | 0.84 | 0.40 | -0.13 | 0.31 | 99 | Y |
| simplest_design | 828 | Q | 0 | estimator | (Intercept) | 0.20 | 0.10 | 2.05 | 0.04 | 0.01 | 0.40 | 99 | Y |
| simplest_design | 829 | Q | 0 | estimator | (Intercept) | 0.10 | 0.09 | 1.12 | 0.27 | -0.07 | 0.26 | 99 | Y |
| simplest_design | 830 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.41 | 0.68 | -0.16 | 0.25 | 99 | Y |
| simplest_design | 831 | Q | 0 | estimator | (Intercept) | 0.08 | 0.09 | 0.85 | 0.40 | -0.11 | 0.27 | 99 | Y |
| simplest_design | 832 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.18 | 0.24 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 833 | Q | 0 | estimator | (Intercept) | -0.04 | 0.13 | -0.32 | 0.75 | -0.29 | 0.21 | 99 | Y |
| simplest_design | 834 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.16 | 0.87 | -0.20 | 0.24 | 99 | Y |
| simplest_design | 835 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.07 | 0.94 | -0.19 | 0.18 | 99 | Y |
| simplest_design | 836 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.33 | 0.74 | -0.25 | 0.18 | 99 | Y |
| simplest_design | 837 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.61 | 0.11 | -0.35 | 0.04 | 99 | Y |
| simplest_design | 838 | Q | 0 | estimator | (Intercept) | -0.21 | 0.11 | -1.81 | 0.07 | -0.43 | 0.02 | 99 | Y |
| simplest_design | 839 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.54 | 0.59 | -0.25 | 0.14 | 99 | Y |
| simplest_design | 840 | Q | 0 | estimator | (Intercept) | 0.04 | 0.11 | 0.38 | 0.70 | -0.18 | 0.26 | 99 | Y |
| simplest_design | 841 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.24 | 0.81 | -0.21 | 0.17 | 99 | Y |
| simplest_design | 842 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.76 | 0.45 | -0.27 | 0.12 | 99 | Y |
| simplest_design | 843 | Q | 0 | estimator | (Intercept) | 0.16 | 0.10 | 1.51 | 0.13 | -0.05 | 0.37 | 99 | Y |
| simplest_design | 844 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.45 | 0.65 | -0.26 | 0.16 | 99 | Y |
| simplest_design | 845 | Q | 0 | estimator | (Intercept) | -0.12 | 0.11 | -1.07 | 0.29 | -0.34 | 0.10 | 99 | Y |
| simplest_design | 846 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.29 | 0.20 | -0.07 | 0.33 | 99 | Y |
| simplest_design | 847 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.17 | 0.87 | -0.18 | 0.21 | 99 | Y |
| simplest_design | 848 | Q | 0 | estimator | (Intercept) | -0.03 | 0.11 | -0.31 | 0.75 | -0.25 | 0.18 | 99 | Y |
| simplest_design | 849 | Q | 0 | estimator | (Intercept) | 0.13 | 0.11 | 1.26 | 0.21 | -0.08 | 0.34 | 99 | Y |
| simplest_design | 850 | Q | 0 | estimator | (Intercept) | 0.26 | 0.10 | 2.58 | 0.01 | 0.06 | 0.45 | 99 | Y |
| simplest_design | 851 | Q | 0 | estimator | (Intercept) | -0.16 | 0.11 | -1.51 | 0.13 | -0.38 | 0.05 | 99 | Y |
| simplest_design | 852 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.56 | 0.58 | -0.14 | 0.24 | 99 | Y |
| simplest_design | 853 | Q | 0 | estimator | (Intercept) | 0.07 | 0.09 | 0.80 | 0.43 | -0.11 | 0.26 | 99 | Y |
| simplest_design | 854 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.36 | 0.72 | -0.25 | 0.17 | 99 | Y |
| simplest_design | 855 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.73 | 0.47 | -0.28 | 0.13 | 99 | Y |
| simplest_design | 856 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.54 | 0.59 | -0.14 | 0.24 | 99 | Y |
| simplest_design | 857 | Q | 0 | estimator | (Intercept) | -0.10 | 0.11 | -0.89 | 0.38 | -0.31 | 0.12 | 99 | Y |
| simplest_design | 858 | Q | 0 | estimator | (Intercept) | 0.11 | 0.11 | 1.03 | 0.30 | -0.10 | 0.32 | 99 | Y |
| simplest_design | 859 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.17 | 0.86 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 860 | Q | 0 | estimator | (Intercept) | 0.06 | 0.11 | 0.58 | 0.57 | -0.15 | 0.27 | 99 | Y |
| simplest_design | 861 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.88 | 0.38 | -0.29 | 0.11 | 99 | Y |
| simplest_design | 862 | Q | 0 | estimator | (Intercept) | -0.12 | 0.11 | -1.11 | 0.27 | -0.33 | 0.09 | 99 | Y |
| simplest_design | 863 | Q | 0 | estimator | (Intercept) | 0.02 | 0.08 | 0.27 | 0.79 | -0.15 | 0.19 | 99 | Y |
| simplest_design | 864 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.86 | 0.39 | -0.30 | 0.12 | 99 | Y |
| simplest_design | 865 | Q | 0 | estimator | (Intercept) | 0.08 | 0.09 | 0.84 | 0.40 | -0.11 | 0.26 | 99 | Y |
| simplest_design | 866 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.17 | 0.87 | -0.17 | 0.21 | 99 | Y |
| simplest_design | 867 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.27 | 0.79 | -0.16 | 0.21 | 99 | Y |
| simplest_design | 868 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.59 | 0.56 | -0.27 | 0.15 | 99 | Y |
| simplest_design | 869 | Q | 0 | estimator | (Intercept) | 0.11 | 0.09 | 1.18 | 0.24 | -0.07 | 0.29 | 99 | Y |
| simplest_design | 870 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.58 | 0.56 | -0.24 | 0.13 | 99 | Y |
| simplest_design | 871 | Q | 0 | estimator | (Intercept) | 0.21 | 0.10 | 2.06 | 0.04 | 0.01 | 0.41 | 99 | Y |
| simplest_design | 872 | Q | 0 | estimator | (Intercept) | 0.15 | 0.09 | 1.58 | 0.12 | -0.04 | 0.33 | 99 | Y |
| simplest_design | 873 | Q | 0 | estimator | (Intercept) | 0.19 | 0.09 | 2.08 | 0.04 | 0.01 | 0.37 | 99 | Y |
| simplest_design | 874 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.16 | 0.25 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 875 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.07 | 0.94 | -0.19 | 0.21 | 99 | Y |
| simplest_design | 876 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.02 | 0.31 | -0.10 | 0.30 | 99 | Y |
| simplest_design | 877 | Q | 0 | estimator | (Intercept) | -0.07 | 0.11 | -0.67 | 0.50 | -0.28 | 0.14 | 99 | Y |
| simplest_design | 878 | Q | 0 | estimator | (Intercept) | 0.04 | 0.08 | 0.53 | 0.60 | -0.12 | 0.21 | 99 | Y |
| simplest_design | 879 | Q | 0 | estimator | (Intercept) | 0.13 | 0.11 | 1.17 | 0.24 | -0.09 | 0.34 | 99 | Y |
| simplest_design | 880 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.37 | 0.71 | -0.25 | 0.17 | 99 | Y |
| simplest_design | 881 | Q | 0 | estimator | (Intercept) | 0.08 | 0.10 | 0.84 | 0.40 | -0.11 | 0.27 | 99 | Y |
| simplest_design | 882 | Q | 0 | estimator | (Intercept) | -0.10 | 0.09 | -1.08 | 0.28 | -0.28 | 0.08 | 99 | Y |
| simplest_design | 883 | Q | 0 | estimator | (Intercept) | 0.00 | 0.11 | -0.01 | 0.99 | -0.22 | 0.21 | 99 | Y |
| simplest_design | 884 | Q | 0 | estimator | (Intercept) | -0.21 | 0.10 | -2.16 | 0.03 | -0.40 | -0.02 | 99 | Y |
| simplest_design | 885 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.07 | 0.94 | -0.20 | 0.18 | 99 | Y |
| simplest_design | 886 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.24 | 0.81 | -0.18 | 0.23 | 99 | Y |
| simplest_design | 887 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.81 | 0.42 | -0.27 | 0.12 | 99 | Y |
| simplest_design | 888 | Q | 0 | estimator | (Intercept) | 0.04 | 0.08 | 0.47 | 0.64 | -0.13 | 0.21 | 99 | Y |
| simplest_design | 889 | Q | 0 | estimator | (Intercept) | 0.16 | 0.09 | 1.68 | 0.10 | -0.03 | 0.34 | 99 | Y |
| simplest_design | 890 | Q | 0 | estimator | (Intercept) | 0.01 | 0.10 | 0.09 | 0.93 | -0.19 | 0.20 | 99 | Y |
| simplest_design | 891 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.26 | 0.79 | -0.16 | 0.20 | 99 | Y |
| simplest_design | 892 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.38 | 0.70 | -0.24 | 0.16 | 99 | Y |
| simplest_design | 893 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.22 | 0.82 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 894 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.12 | 0.27 | -0.31 | 0.09 | 99 | Y |
| simplest_design | 895 | Q | 0 | estimator | (Intercept) | 0.06 | 0.10 | 0.61 | 0.54 | -0.13 | 0.25 | 99 | Y |
| simplest_design | 896 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.68 | 0.50 | -0.27 | 0.13 | 99 | Y |
| simplest_design | 897 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.38 | 0.17 | -0.35 | 0.06 | 99 | Y |
| simplest_design | 898 | Q | 0 | estimator | (Intercept) | 0.12 | 0.10 | 1.27 | 0.21 | -0.07 | 0.31 | 99 | Y |
| simplest_design | 899 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.19 | 0.85 | -0.20 | 0.25 | 99 | Y |
| simplest_design | 900 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.89 | 0.38 | -0.11 | 0.28 | 99 | Y |
| simplest_design | 901 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.06 | 0.95 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 902 | Q | 0 | estimator | (Intercept) | -0.11 | 0.11 | -1.06 | 0.29 | -0.32 | 0.10 | 99 | Y |
| simplest_design | 903 | Q | 0 | estimator | (Intercept) | -0.08 | 0.09 | -0.91 | 0.36 | -0.26 | 0.10 | 99 | Y |
| simplest_design | 904 | Q | 0 | estimator | (Intercept) | 0.07 | 0.10 | 0.70 | 0.48 | -0.13 | 0.28 | 99 | Y |
| simplest_design | 905 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.59 | 0.55 | -0.28 | 0.15 | 99 | Y |
| simplest_design | 906 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.33 | 0.74 | -0.16 | 0.22 | 99 | Y |
| simplest_design | 907 | Q | 0 | estimator | (Intercept) | -0.01 | 0.11 | -0.14 | 0.89 | -0.23 | 0.20 | 99 | Y |
| simplest_design | 908 | Q | 0 | estimator | (Intercept) | -0.17 | 0.10 | -1.73 | 0.09 | -0.38 | 0.03 | 99 | Y |
| simplest_design | 909 | Q | 0 | estimator | (Intercept) | -0.06 | 0.11 | -0.52 | 0.60 | -0.28 | 0.16 | 99 | Y |
| simplest_design | 910 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.21 | 0.84 | -0.21 | 0.17 | 99 | Y |
| simplest_design | 911 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.24 | 0.81 | -0.23 | 0.18 | 99 | Y |
| simplest_design | 912 | Q | 0 | estimator | (Intercept) | -0.04 | 0.11 | -0.36 | 0.72 | -0.25 | 0.18 | 99 | Y |
| simplest_design | 913 | Q | 0 | estimator | (Intercept) | 0.17 | 0.09 | 1.80 | 0.08 | -0.02 | 0.36 | 99 | Y |
| simplest_design | 914 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | -0.04 | 0.96 | -0.21 | 0.20 | 99 | Y |
| simplest_design | 915 | Q | 0 | estimator | (Intercept) | -0.05 | 0.10 | -0.52 | 0.61 | -0.25 | 0.15 | 99 | Y |
| simplest_design | 916 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.68 | 0.50 | -0.27 | 0.13 | 99 | Y |
| simplest_design | 917 | Q | 0 | estimator | (Intercept) | 0.20 | 0.09 | 2.15 | 0.03 | 0.02 | 0.39 | 99 | Y |
| simplest_design | 918 | Q | 0 | estimator | (Intercept) | -0.09 | 0.09 | -1.00 | 0.32 | -0.26 | 0.08 | 99 | Y |
| simplest_design | 919 | Q | 0 | estimator | (Intercept) | -0.05 | 0.11 | -0.41 | 0.68 | -0.27 | 0.18 | 99 | Y |
| simplest_design | 920 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.37 | 0.71 | -0.15 | 0.22 | 99 | Y |
| simplest_design | 921 | Q | 0 | estimator | (Intercept) | 0.13 | 0.10 | 1.30 | 0.20 | -0.07 | 0.33 | 99 | Y |
| simplest_design | 922 | Q | 0 | estimator | (Intercept) | -0.04 | 0.10 | -0.36 | 0.72 | -0.24 | 0.17 | 99 | Y |
| simplest_design | 923 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.11 | 0.27 | -0.08 | 0.30 | 99 | Y |
| simplest_design | 924 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.48 | 0.63 | -0.14 | 0.23 | 99 | Y |
| simplest_design | 925 | Q | 0 | estimator | (Intercept) | 0.02 | 0.11 | 0.18 | 0.86 | -0.20 | 0.24 | 99 | Y |
| simplest_design | 926 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.54 | 0.59 | -0.22 | 0.12 | 99 | Y |
| simplest_design | 927 | Q | 0 | estimator | (Intercept) | -0.18 | 0.10 | -1.78 | 0.08 | -0.38 | 0.02 | 99 | Y |
| simplest_design | 928 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.30 | 0.76 | -0.23 | 0.17 | 99 | Y |
| simplest_design | 929 | Q | 0 | estimator | (Intercept) | 0.08 | 0.11 | 0.69 | 0.49 | -0.15 | 0.30 | 99 | Y |
| simplest_design | 930 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.25 | 0.80 | -0.23 | 0.18 | 99 | Y |
| simplest_design | 931 | Q | 0 | estimator | (Intercept) | 0.04 | 0.11 | 0.39 | 0.70 | -0.18 | 0.27 | 99 | Y |
| simplest_design | 932 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.20 | 0.84 | -0.22 | 0.18 | 99 | Y |
| simplest_design | 933 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.26 | 0.80 | -0.16 | 0.21 | 99 | Y |
| simplest_design | 934 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.48 | 0.63 | -0.14 | 0.23 | 99 | Y |
| simplest_design | 935 | Q | 0 | estimator | (Intercept) | 0.02 | 0.10 | 0.22 | 0.82 | -0.17 | 0.21 | 99 | Y |
| simplest_design | 936 | Q | 0 | estimator | (Intercept) | 0.24 | 0.10 | 2.42 | 0.02 | 0.04 | 0.43 | 99 | Y |
| simplest_design | 937 | Q | 0 | estimator | (Intercept) | -0.21 | 0.09 | -2.30 | 0.02 | -0.39 | -0.03 | 99 | Y |
| simplest_design | 938 | Q | 0 | estimator | (Intercept) | -0.28 | 0.10 | -2.86 | 0.01 | -0.47 | -0.08 | 99 | Y |
| simplest_design | 939 | Q | 0 | estimator | (Intercept) | 0.06 | 0.12 | 0.52 | 0.60 | -0.17 | 0.29 | 99 | Y |
| simplest_design | 940 | Q | 0 | estimator | (Intercept) | 0.04 | 0.10 | 0.43 | 0.67 | -0.15 | 0.23 | 99 | Y |
| simplest_design | 941 | Q | 0 | estimator | (Intercept) | 0.17 | 0.09 | 1.83 | 0.07 | -0.01 | 0.36 | 99 | Y |
| simplest_design | 942 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.24 | 0.22 | -0.31 | 0.07 | 99 | Y |
| simplest_design | 943 | Q | 0 | estimator | (Intercept) | -0.16 | 0.09 | -1.85 | 0.07 | -0.33 | 0.01 | 99 | Y |
| simplest_design | 944 | Q | 0 | estimator | (Intercept) | 0.16 | 0.10 | 1.55 | 0.13 | -0.05 | 0.37 | 99 | Y |
| simplest_design | 945 | Q | 0 | estimator | (Intercept) | 0.13 | 0.11 | 1.18 | 0.24 | -0.09 | 0.36 | 99 | Y |
| simplest_design | 946 | Q | 0 | estimator | (Intercept) | -0.10 | 0.10 | -0.97 | 0.33 | -0.30 | 0.10 | 99 | Y |
| simplest_design | 947 | Q | 0 | estimator | (Intercept) | 0.16 | 0.10 | 1.56 | 0.12 | -0.04 | 0.36 | 99 | Y |
| simplest_design | 948 | Q | 0 | estimator | (Intercept) | 0.18 | 0.10 | 1.84 | 0.07 | -0.01 | 0.37 | 99 | Y |
| simplest_design | 949 | Q | 0 | estimator | (Intercept) | 0.06 | 0.09 | 0.62 | 0.54 | -0.13 | 0.25 | 99 | Y |
| simplest_design | 950 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.22 | 0.83 | -0.20 | 0.16 | 99 | Y |
| simplest_design | 951 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.67 | 0.50 | -0.28 | 0.14 | 99 | Y |
| simplest_design | 952 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.72 | 0.47 | -0.26 | 0.12 | 99 | Y |
| simplest_design | 953 | Q | 0 | estimator | (Intercept) | 0.18 | 0.10 | 1.74 | 0.08 | -0.02 | 0.38 | 99 | Y |
| simplest_design | 954 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.43 | 0.16 | -0.35 | 0.06 | 99 | Y |
| simplest_design | 955 | Q | 0 | estimator | (Intercept) | -0.12 | 0.10 | -1.28 | 0.20 | -0.31 | 0.07 | 99 | Y |
| simplest_design | 956 | Q | 0 | estimator | (Intercept) | -0.26 | 0.10 | -2.54 | 0.01 | -0.45 | -0.06 | 99 | Y |
| simplest_design | 957 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | -0.04 | 0.97 | -0.18 | 0.17 | 99 | Y |
| simplest_design | 958 | Q | 0 | estimator | (Intercept) | -0.05 | 0.09 | -0.53 | 0.60 | -0.24 | 0.14 | 99 | Y |
| simplest_design | 959 | Q | 0 | estimator | (Intercept) | -0.11 | 0.09 | -1.23 | 0.22 | -0.29 | 0.07 | 99 | Y |
| simplest_design | 960 | Q | 0 | estimator | (Intercept) | -0.25 | 0.11 | -2.37 | 0.02 | -0.47 | -0.04 | 99 | Y |
| simplest_design | 961 | Q | 0 | estimator | (Intercept) | 0.00 | 0.09 | -0.04 | 0.97 | -0.19 | 0.18 | 99 | Y |
| simplest_design | 962 | Q | 0 | estimator | (Intercept) | -0.03 | 0.10 | -0.27 | 0.78 | -0.23 | 0.18 | 99 | Y |
| simplest_design | 963 | Q | 0 | estimator | (Intercept) | -0.12 | 0.11 | -1.10 | 0.28 | -0.33 | 0.10 | 99 | Y |
| simplest_design | 964 | Q | 0 | estimator | (Intercept) | -0.06 | 0.10 | -0.60 | 0.55 | -0.26 | 0.14 | 99 | Y |
| simplest_design | 965 | Q | 0 | estimator | (Intercept) | 0.09 | 0.10 | 0.83 | 0.41 | -0.12 | 0.29 | 99 | Y |
| simplest_design | 966 | Q | 0 | estimator | (Intercept) | -0.10 | 0.09 | -1.10 | 0.27 | -0.27 | 0.08 | 99 | Y |
| simplest_design | 967 | Q | 0 | estimator | (Intercept) | -0.01 | 0.09 | -0.13 | 0.90 | -0.20 | 0.17 | 99 | Y |
| simplest_design | 968 | Q | 0 | estimator | (Intercept) | 0.10 | 0.10 | 1.04 | 0.30 | -0.09 | 0.30 | 99 | Y |
| simplest_design | 969 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.72 | 0.47 | -0.28 | 0.13 | 99 | Y |
| simplest_design | 970 | Q | 0 | estimator | (Intercept) | -0.02 | 0.09 | -0.20 | 0.85 | -0.20 | 0.17 | 99 | Y |
| simplest_design | 971 | Q | 0 | estimator | (Intercept) | -0.09 | 0.10 | -0.91 | 0.36 | -0.29 | 0.11 | 99 | Y |
| simplest_design | 972 | Q | 0 | estimator | (Intercept) | 0.05 | 0.09 | 0.54 | 0.59 | -0.13 | 0.23 | 99 | Y |
| simplest_design | 973 | Q | 0 | estimator | (Intercept) | 0.03 | 0.11 | 0.30 | 0.76 | -0.19 | 0.25 | 99 | Y |
| simplest_design | 974 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | -0.05 | 0.96 | -0.20 | 0.19 | 99 | Y |
| simplest_design | 975 | Q | 0 | estimator | (Intercept) | -0.11 | 0.09 | -1.15 | 0.25 | -0.29 | 0.08 | 99 | Y |
| simplest_design | 976 | Q | 0 | estimator | (Intercept) | 0.24 | 0.09 | 2.55 | 0.01 | 0.05 | 0.43 | 99 | Y |
| simplest_design | 977 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.80 | 0.43 | -0.28 | 0.12 | 99 | Y |
| simplest_design | 978 | Q | 0 | estimator | (Intercept) | 0.04 | 0.11 | 0.42 | 0.68 | -0.16 | 0.25 | 99 | Y |
| simplest_design | 979 | Q | 0 | estimator | (Intercept) | 0.00 | 0.10 | 0.05 | 0.96 | -0.20 | 0.21 | 99 | Y |
| simplest_design | 980 | Q | 0 | estimator | (Intercept) | 0.18 | 0.10 | 1.74 | 0.08 | -0.02 | 0.38 | 99 | Y |
| simplest_design | 981 | Q | 0 | estimator | (Intercept) | -0.04 | 0.09 | -0.45 | 0.65 | -0.21 | 0.13 | 99 | Y |
| simplest_design | 982 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.04 | 0.30 | -0.31 | 0.10 | 99 | Y |
| simplest_design | 983 | Q | 0 | estimator | (Intercept) | -0.10 | 0.11 | -0.94 | 0.35 | -0.31 | 0.11 | 99 | Y |
| simplest_design | 984 | Q | 0 | estimator | (Intercept) | 0.02 | 0.09 | 0.22 | 0.83 | -0.16 | 0.20 | 99 | Y |
| simplest_design | 985 | Q | 0 | estimator | (Intercept) | -0.08 | 0.10 | -0.81 | 0.42 | -0.27 | 0.11 | 99 | Y |
| simplest_design | 986 | Q | 0 | estimator | (Intercept) | 0.11 | 0.10 | 1.12 | 0.26 | -0.09 | 0.31 | 99 | Y |
| simplest_design | 987 | Q | 0 | estimator | (Intercept) | 0.03 | 0.10 | 0.28 | 0.78 | -0.17 | 0.22 | 99 | Y |
| simplest_design | 988 | Q | 0 | estimator | (Intercept) | -0.11 | 0.10 | -1.16 | 0.25 | -0.31 | 0.08 | 99 | Y |
| simplest_design | 989 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.55 | 0.59 | -0.14 | 0.25 | 99 | Y |
| simplest_design | 990 | Q | 0 | estimator | (Intercept) | -0.13 | 0.11 | -1.26 | 0.21 | -0.34 | 0.08 | 99 | Y |
| simplest_design | 991 | Q | 0 | estimator | (Intercept) | -0.06 | 0.09 | -0.67 | 0.50 | -0.24 | 0.12 | 99 | Y |
| simplest_design | 992 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.35 | 0.18 | -0.34 | 0.06 | 99 | Y |
| simplest_design | 993 | Q | 0 | estimator | (Intercept) | 0.05 | 0.10 | 0.53 | 0.60 | -0.15 | 0.26 | 99 | Y |
| simplest_design | 994 | Q | 0 | estimator | (Intercept) | -0.07 | 0.10 | -0.68 | 0.50 | -0.28 | 0.14 | 99 | Y |
| simplest_design | 995 | Q | 0 | estimator | (Intercept) | 0.14 | 0.11 | 1.29 | 0.20 | -0.07 | 0.35 | 99 | Y |
| simplest_design | 996 | Q | 0 | estimator | (Intercept) | -0.01 | 0.10 | -0.13 | 0.89 | -0.20 | 0.18 | 99 | Y |
| simplest_design | 997 | Q | 0 | estimator | (Intercept) | -0.02 | 0.10 | -0.23 | 0.82 | -0.23 | 0.18 | 99 | Y |
| simplest_design | 998 | Q | 0 | estimator | (Intercept) | 0.21 | 0.11 | 1.97 | 0.05 | 0.00 | 0.42 | 99 | Y |
| simplest_design | 999 | Q | 0 | estimator | (Intercept) | 0.16 | 0.10 | 1.67 | 0.10 | -0.03 | 0.35 | 99 | Y |
| simplest_design | 1000 | Q | 0 | estimator | (Intercept) | 0.13 | 0.11 | 1.17 | 0.24 | -0.09 | 0.34 | 99 | Y |
Once you have simulated many times you can “diagnose”.
This is the next topic
Once you have simulated many times you can “diagnose”.
For instance we can ask about bias: the average difference between the estimand and the estimate:
| mean_estimate | mean_estimand | bias |
|---|---|---|
| 0 | 0 | 0 |
diagnose_design()diagnose_design() does this in one step for a set of common “diagnosands”:
| Design | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|
| simplest_design | 500 | 0.00 | -0.00 | -0.00 | 0.10 | 0.10 | 0.05 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.01) | (0.01) |
The diagnosis object is also a list; of class diagnosis
| design | sim_ID | inquiry | estimand | estimator | term | estimate | std.error | statistic | p.value | conf.low | conf.high | df | outcome |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| simplest_design | 1 | Q | 0 | estimator | (Intercept) | 0.03 | 0.09 | 0.31 | 0.76 | -0.16 | 0.21 | 99 | Y |
| simplest_design | 2 | Q | 0 | estimator | (Intercept) | 0.10 | 0.09 | 1.07 | 0.29 | -0.09 | 0.29 | 99 | Y |
| simplest_design | 3 | Q | 0 | estimator | (Intercept) | -0.16 | 0.10 | -1.54 | 0.13 | -0.37 | 0.05 | 99 | Y |
| simplest_design | 4 | Q | 0 | estimator | (Intercept) | -0.08 | 0.11 | -0.72 | 0.48 | -0.30 | 0.14 | 99 | Y |
| simplest_design | 5 | Q | 0 | estimator | (Intercept) | -0.14 | 0.10 | -1.34 | 0.18 | -0.34 | 0.07 | 99 | Y |
| simplest_design | 6 | Q | 0 | estimator | (Intercept) | -0.08 | 0.09 | -0.90 | 0.37 | -0.26 | 0.10 | 99 | Y |
| design | inquiry | estimator | outcome | term | mean_estimand | se(mean_estimand) | mean_estimate | se(mean_estimate) | bias | se(bias) | sd_estimate | se(sd_estimate) | rmse | se(rmse) | power | se(power) | coverage | se(coverage) | n_sims |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| simplest_design | Q | estimator | Y | (Intercept) | 0 | 0 | 0 | 0 | 0 | 0 | 0.1 | 0 | 0.1 | 0 | 0.05 | 0.01 | 0.95 | 0.01 | 500 |
| design | bootstrap_id | inquiry | estimator | outcome | term | mean_estimand | mean_estimate | bias | sd_estimate | rmse | power | coverage |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| simplest_design | 1 | Q | estimator | Y | (Intercept) | 0 | 0.00 | 0.00 | 0.1 | 0.10 | 0.05 | 0.95 |
| simplest_design | 2 | Q | estimator | Y | (Intercept) | 0 | -0.01 | -0.01 | 0.1 | 0.11 | 0.06 | 0.94 |
| simplest_design | 3 | Q | estimator | Y | (Intercept) | 0 | -0.01 | -0.01 | 0.1 | 0.10 | 0.05 | 0.95 |
| simplest_design | 4 | Q | estimator | Y | (Intercept) | 0 | -0.01 | -0.01 | 0.1 | 0.10 | 0.05 | 0.95 |
| simplest_design | 5 | Q | estimator | Y | (Intercept) | 0 | 0.00 | 0.00 | 0.1 | 0.10 | 0.05 | 0.95 |
| simplest_design | 6 | Q | estimator | Y | (Intercept) | 0 | 0.00 | 0.00 | 0.1 | 0.10 | 0.05 | 0.95 |
The bootstraps dataframe is produced by resampling from the simulations dataframe and producing a diagnosis dataframe from each resampling.
This lets us generate estimates of uncertainty around our diagnosands.
It can be controlled thus:
It’s reshapeable: as a tidy dataframe, ready for graphing
| design | inquiry | estimator | outcome | term | diagnosand | estimate | std.error | conf.low | conf.high |
|---|---|---|---|---|---|---|---|---|---|
| simplest_design | Q | estimator | Y | (Intercept) | mean_estimand | 0.00 | 0.00 | 0.00 | 0.00 |
| simplest_design | Q | estimator | Y | (Intercept) | mean_estimate | 0.00 | 0.00 | -0.01 | 0.00 |
| simplest_design | Q | estimator | Y | (Intercept) | bias | 0.00 | 0.00 | -0.01 | 0.00 |
| simplest_design | Q | estimator | Y | (Intercept) | sd_estimate | 0.10 | 0.00 | 0.10 | 0.11 |
| simplest_design | Q | estimator | Y | (Intercept) | rmse | 0.10 | 0.00 | 0.10 | 0.11 |
| simplest_design | Q | estimator | Y | (Intercept) | power | 0.05 | 0.01 | 0.03 | 0.07 |
| simplest_design | Q | estimator | Y | (Intercept) | coverage | 0.95 | 0.01 | 0.93 | 0.97 |
It’s reshapeable: as a tidy dataframe, ready for graphing
Or turn into a formatted table:
| Design | Inquiry | Estimator | Outcome | Term | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| simplest_design | Q | estimator | Y | (Intercept) | 500 | 0.00 | -0.00 | -0.00 | 0.10 | 0.10 | 0.05 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.01) | (0.01) |
Diagnosis alerts to problems in a design. Consider the following simple alternative design.
Here we define the inquiry as the sample average \(Y\) (instead of the population mean). But otherwise things stay the same.
What do we think of this design?
Here is the diagnosis
| Design | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|
| simplest_design_2 | 500 | -0.00 | -0.00 | 0.00 | 0.10 | 0.00 | 0.04 | 1.00 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.01) | (0.00) |
You can diagnose multiple designs or a list of designs
list(dum = simplest_design, dee = simplest_design) |>
diagnose_design(sims = 5) |>
reshape_diagnosis() |>
kable() |>
kable_styling(font_size = 20)| Design | Inquiry | Estimator | Outcome | Term | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dum | Q | estimator | Y | (Intercept) | 5 | 0.00 | -0.01 | -0.01 | 0.08 | 0.07 | 0.00 | 1.00 |
| (0.00) | (0.03) | (0.03) | (0.01) | (0.01) | (0.00) | (0.00) | ||||||
| dee | Q | estimator | Y | (Intercept) | 5 | 0.00 | 0.02 | 0.02 | 0.12 | 0.11 | 0.00 | 1.00 |
| (0.00) | (0.05) | (0.05) | (0.03) | (0.02) | (0.00) | (0.00) |
Redesign is the process of taking a design and modifying it in some way.
There are a few ways to do this:
replace_step, insert_step or delete_stepredesignwe will focus on the third approach
A design parameter is a modifiable quantity of a design.
These quantities are objects that were in your global environment when you made your design, get referred to explicitly in your design, and got scooped up when the design was formed.
In our simplest design above we had a fixed N, but we could make N a modifiable quantity like this:
Note that N is defined in memory; and it gets called in one of the steps. It has now become a parameter of the design and it can be modified using redesign.
Here is a version of the design with N = 200:
Here is a list of three different designs with different Ns.
The good thing here is that it is now easy to diagnose over multiple designs and compare diagnoses. The parameter names then end up in the diagnosis_df
Consider this:
Then:
Output:
| N | m | diagnosand | estimate | std.error | conf.low | conf.high |
|---|---|---|---|---|---|---|
| 100 | 0.0 | mean_estimand | 0.00 | 0.00 | 0.00 | 0.00 |
| 100 | 0.0 | mean_estimate | 0.00 | 0.00 | -0.01 | 0.01 |
| 100 | 0.0 | bias | 0.00 | 0.00 | -0.01 | 0.01 |
| 100 | 0.0 | sd_estimate | 0.10 | 0.00 | 0.10 | 0.11 |
| 200 | 0.0 | mean_estimand | 0.00 | 0.00 | 0.00 | 0.00 |
| 200 | 0.0 | mean_estimate | 0.00 | 0.00 | -0.01 | 0.00 |
| 200 | 0.1 | mean_estimand | 0.10 | 0.00 | 0.10 | 0.10 |
| 200 | 0.1 | mean_estimate | 0.10 | 0.00 | 0.09 | 0.10 |
| 300 | 0.2 | bias | 0.00 | 0.00 | 0.00 | 0.00 |
| 300 | 0.2 | sd_estimate | 0.06 | 0.00 | 0.05 | 0.06 |
| 300 | 0.2 | rmse | 0.06 | 0.00 | 0.05 | 0.06 |
| 300 | 0.2 | power | 0.93 | 0.01 | 0.91 | 0.95 |
| 300 | 0.2 | coverage | 0.95 | 0.01 | 0.92 | 0.97 |
Graphing after redesign is especially easy:
Power depends on N and m, rmse depends on N only
What can you do with a design once you have it?
We motivate with a slightly more complex experimental design (more on the components of this later)
| ID | U | Y_Z_0 | Y_Z_1 | Z | Y |
|---|---|---|---|---|---|
| 001 | -1.2516446 | -1.2516446 | -0.2516446 | 0 | -1.2516446 |
| 002 | -1.4308705 | -1.4308705 | -0.4308705 | 1 | -0.4308705 |
| 003 | 0.8499598 | 0.8499598 | 1.8499598 | 0 | 0.8499598 |
| 004 | 0.2390656 | 0.2390656 | 1.2390656 | 0 | 0.2390656 |
| 005 | -0.5418315 | -0.5418315 | 0.4581685 | 1 | 0.4581685 |
| 006 | 2.3695282 | 2.3695282 | 3.3695282 | 1 | 3.3695282 |
Play with the data:
Using your actual data:
| design | sim_ID | inquiry | estimand | estimator | term | estimate | std.error | statistic | p.value | conf.low | conf.high | df | outcome |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| design | 1 | ate | 1 | estimator | Z | 0.80 | 0.19 | 4.10 | 0 | 0.41 | 1.18 | 98 | Y |
| design | 2 | ate | 1 | estimator | Z | 0.95 | 0.20 | 4.80 | 0 | 0.56 | 1.35 | 98 | Y |
| design | 3 | ate | 1 | estimator | Z | 1.25 | 0.21 | 5.96 | 0 | 0.83 | 1.66 | 98 | Y |
| Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|
| 0.99 | -0.01 | 0.22 | 0.22 | 1.00 | 0.93 |
| (0.02) | (0.02) | (0.01) | (0.01) | (0.00) | (0.02) |
| diagnosand | mean_1 | mean_2 | mean_difference | conf.low | conf.high |
|---|---|---|---|---|---|
| mean_estimand | 0.50 | 0.50 | 0.00 | 0.00 | 0.00 |
| mean_estimate | 0.48 | 0.50 | 0.02 | -0.01 | 0.04 |
| bias | -0.02 | 0.00 | 0.02 | -0.01 | 0.04 |
| sd_estimate | 0.28 | 0.20 | -0.08 | -0.10 | -0.06 |
| rmse | 0.28 | 0.20 | -0.08 | -0.10 | -0.06 |
| power | 0.38 | 0.71 | 0.32 | 0.26 | 0.37 |
| coverage | 0.97 | 0.96 | -0.01 | -0.04 | 0.01 |
DeclareDesign: A deeper diveWe start with a simple experimental design with all four elements of MIDA and then show ways to extend.
fabricatr package (and others)randomizr package (and others)estimatr package (and others)New elements:
declare_model can be used much like mutate with multiple columns created in sequencepotential_outcomes function is a special function that creates potential outcome columns for different values of Zreveal_outcome to reveal the outcome; Z and Y are defaultlm_robust is default)e.g. If you sample before defining the inquiry you get a different inquiry to if you sample after you define the inquiry
e.g. If you sample before defining the inquiry you get a different inquiry to if you sample after you define the inquiry
You can generate hierarchical data like this:
You can generate hierarchical data like this:
You can generate panel data like this:
M <-
declare_model(
countries = add_level(
N = 196,
country_shock = rnorm(N)
),
years = add_level(
N = 100,
time_trend = 1:N,
year_shock = runif(N, 1, 10),
nest = FALSE
),
observation = cross_levels(
by = join_using(countries, years),
observation_shock = rnorm(N),
Y = 0.01 * time_trend + country_shock + year_shock + observation_shock
)
)You can generate panel data like this:
| countries | country_shock | years | time_trend | year_shock | observation | observation_shock | Y |
|---|---|---|---|---|---|---|---|
| 001 | 0.40 | 001 | 1 | 5.22 | 00001 | -0.04 | 5.59 |
| 002 | -0.86 | 001 | 1 | 5.22 | 00002 | -0.01 | 4.36 |
| 003 | 0.55 | 001 | 1 | 5.22 | 00003 | 0.31 | 6.09 |
| 004 | 1.10 | 001 | 1 | 5.22 | 00004 | 1.27 | 7.60 |
| 005 | 1.51 | 001 | 1 | 5.22 | 00005 | 0.26 | 7.00 |
| 006 | -1.21 | 001 | 1 | 5.22 | 00006 | -0.13 | 3.89 |
You can repeat steps and play with the order, always conscious of the direction of the pipe
design <-
declare_model(N = N, X = rep(0:1, N/2)) +
declare_model(U = rnorm(N), potential_outcomes(Y ~ b * Z * X + U)) +
declare_assignment(Z = block_ra(blocks = X), Y = reveal_outcomes(Y ~ Z)) +
declare_inquiry(ate = mean(Y_Z_1 - Y_Z_0)) +
declare_inquiry(cate = mean(Y_Z_1[X==0] - Y_Z_0[X==0])) +
declare_estimator(Y ~ Z, inquiry = "ate", label = "ols") +
declare_estimator(Y ~ Z*X, inquiry = "cate", label = "fe")Many causal inquiries are simple summaries of potential outcomes:
| Inquiry | Units | Code |
|---|---|---|
| Average treatment effect in a finite population (PATE) | Units in the population | mean(Y_D_1 - Y_D_0) |
| Conditional average treatment effect (CATE) for X = 1 | Units for whom X = 1 | mean(Y_D_1[X == 1] - Y_D_0[X == 1]) |
| Complier average causal effect (CACE) | Complier units | mean(Y_D_1[D_Z_1 > D_Z_0] - Y_D_0[D_Z_1 > D_Z_0]) |
| Causal interactions of \(D_1\) and \(D_2\) | Units in the population | mean((Y_D1_1_D2_1 - Y_D1_0_D2_1) - (Y_D1_1_D2_0 - Y_D1_0_D2_0)) |
Generating potential outcomes columns gets you far
Often though we need to define inquiries as a function of continuous variables. For this generating a potential outcomes function can make life easier. This helps for:
Here is an example of using functions to define complex counterfactuals:
f_M <- function(X, UM) 1*(UM < X)
f_Y <- function(X, M, UY) X + M - .4*X*M + UY
design <-
declare_model(N = 100,
X = simple_rs(N),
UM = runif(N),
UY = rnorm(N),
M = f_M(X, UM),
Y = f_Y(X, M, UY)) +
declare_inquiry(Q1 = mean(f_Y(1, f_M(0, UM), UY) - f_Y(0, f_M(0, UM), UY)))
design |> draw_estimands() |> kable() |> kable_styling(font_size = 20)| inquiry | estimand |
|---|---|
| Q1 | 1 |
Here is an example of using functions to define effects of continuous treatments.
f_Y <- function(X, UY) X - .25*X^2 + UY
design <-
declare_model(N = 100,
X = rnorm(N),
UY = rnorm(N),
Y = f_Y(X, UY)) +
declare_inquiry(
Q1 = mean(f_Y(X+1, UY) - f_Y(X, UY)),
Q2 = mean(f_Y(1, UY) - f_Y(0, UY)),
Q3 = (lm_robust(Y ~ X)|> tidy())[2,2]
)
design |> draw_estimands() |> kable() |> kable_styling(font_size = 20)| inquiry | estimand |
|---|---|
| Q1 | 0.7467501 |
| Q2 | 0.7500000 |
| Q3 | 0.9864264 |
which one is the ATE?
The randomizr package has a set of functions for different types of block and cluster assignments.
simple_ra(N = 100, prob = 0.25)complete_ra(N = 100, m = 40)block_ra(blocks = regions)cluster_ra(clusters = households) * Block-and-cluster assignment: Cluster random assignment within blocks of clusters block_and_cluster_ra(blocks = regions, clusters = villages)You can combine these in various ways. For examples with saturation random assignment first clusters are assigned to a saturation level, then units within clusters are assigned to treatment conditions according to the saturation level:
By default declare_estimates() assumes you are interested in the first term after the constant from the output of an estimation procedure.
But you can say what you are interested in directly using term and you can also associate different terms with different quantities of interest using inquiry.
design <-
declare_model(
N = 100,
X1 = rnorm(N),
X2 = rnorm(N),
X3 = rnorm(N),
Y = X1 - X2 + X3 + rnorm(N)
) +
declare_inquiries(ate_2 = -1, ate_3 = 1) +
declare_estimator(Y ~ X1 + X2 + X3,
term = c("X2", "X3"),
inquiry = c("ate_2", "ate_3"))
design |> run_design() |> kable(digits = 2) |> kable_styling(font_size = 20)| inquiry | estimand | term | estimator | estimate | std.error | statistic | p.value | conf.low | conf.high | df | outcome |
|---|---|---|---|---|---|---|---|---|---|---|---|
| ate_2 | -1 | X2 | estimator | -1.04 | 0.10 | -10.55 | 0 | -1.23 | -0.84 | 96 | Y |
| ate_3 | 1 | X3 | estimator | 1.02 | 0.08 | 12.17 | 0 | 0.85 | 1.18 | 96 | Y |
Sometimes it can be confusing what the names of a term is but you can figure this by running the estimation strategy directly. Here’s an example where the names of a term might be confusing.
lm_robust(Y ~ A*B,
data = data.frame(A = rep(c("a", "b"), 3),
B = rep(c("p", "q"), each = 3),
Y = rnorm(6))) |>
coef() |> kable() |> kable_styling(font_size = 20)| x | |
|---|---|
| (Intercept) | 0.3094024 |
| Ab | -0.6618648 |
| Bq | -0.5741195 |
| Ab:Bq | -0.2054769 |
The names as they appear in the output here is the name of the term that declare_estimator will look for.
DeclareDesign works natively with estimatr but you you can use whatever packages you like. You do have to make sure though that estimatr gets as input a nice tidy dataframe of estimates, and that might require some tidying.
design <-
declare_model(N = 1000, U = runif(N),
potential_outcomes(Y ~ as.numeric(U < .5 + Z/3))) +
declare_assignment(Z = simple_ra(N), Y = reveal_outcomes(Y ~ Z)) +
declare_inquiry(ate = mean(Y_Z_1 - Y_Z_0)) +
declare_estimator(Y ~ Z, inquiry = "ate",
.method = glm,
family = binomial(link = "probit"))Note that we passed additional arguments to glm; that’s easy.
It’s not a good design though. Just look at the diagnosis:
if(run)
diagnose_design(design) |> write_rds("saved/probit.rds")
read_rds("saved/probit.rds") |>
reshape_diagnosis() |>
kable() |>
kable_styling(font_size = 20)| Design | Inquiry | Estimator | Term | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|---|---|---|
| design | ate | estimator | Z | 500 | 0.33 | 0.97 | 0.64 | 0.09 | 0.64 | 1.00 | 0.00 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) |
Why is it so terrible?
Because the probit estimate does not target the ATE directly; you need to do more work to get there.
You essentially have to write a function to get the estimates, calculate the quantity of interest and other stats, and turn these into a nice dataframe.
Luckily you can use the margins package with tidy to create a .summary function which you can pass to declare_estimator to do all this for you
if(run)
diagnose_design(design) |> write_rds("saved/probit_2.rds")
read_rds("saved/probit_2.rds") |> reshape_diagnosis() |> kable() |>
kable_styling(font_size = 20)| Design | Inquiry | Estimator | Term | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|---|---|---|
| design | ate | estimator | Z | 500 | 0.33 | 0.97 | 0.64 | 0.09 | 0.64 | 1.00 | 0.00 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | |||||
| design | ate | margins | Z | 500 | 0.33 | 0.31 | -0.02 | 0.02 | 0.03 | 1.00 | 0.90 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.01) |
Much better
mean_se = mean(std.error)
type_s_rate = mean((sign(estimate) != sign(estimand))[p.value <= alpha])
exaggeration_ratio = mean((estimate/estimand)[p.value <= alpha])
var_estimate = pop.var(estimate)
mean_var_hat = mean(std.error^2)
prop_pos_sig = estimate > 0 & p.value <= alpha
mean_ci_length = mean(conf.high - conf.low)my_diagnosands <-
declare_diagnosands(median_bias = median(estimate - estimand))
diagnose_design(simplest_design, diagnosands = my_diagnosands, sims = 10) |>
reshape_diagnosis() |> kable() |> kable_styling(font_size = 20)| Design | Inquiry | Estimator | Outcome | Term | N Sims | Median Bias |
|---|---|---|---|---|---|---|
| simplest_design | Q | estimator | Y | (Intercept) | 10 | -0.01 |
| (0.02) |
You can partition the simulations data frame into groups before calculating diagnosands.
| Design | Significant | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|---|
| design_1 | FALSE | 474 | 0.00 | -0.00 | -0.00 | 0.09 | 0.09 | 0.00 | 1.00 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | |||
| design_1 | TRUE | 26 | 0.00 | -0.02 | -0.02 | 0.23 | 0.23 | 1.00 | 0.00 |
| (0.00) | (0.04) | (0.04) | (0.01) | (0.01) | (0.00) | (0.00) |
Note especially the mean estimate, the power, the coverage, the RMSE, and the bias. (Bias is not large because we have both under and over estimates)
Consider for instance this sampling design:
Compare these two diagnoses:
| diagnosis | N Sims | Mean Estimand | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|---|---|
| diagnosis_1 | 5000 | 1.00 | 1.00 | -0.00 | 1.01 | 0.90 | 0.17 | 0.97 |
| diagnosis_1 | (0.01) | (0.01) | (0.01) | (0.01) | (0.01) | (0.01) | (0.00) | |
| diagnosis_2 | 5000 | 1.22 | 1.22 | -0.00 | 0.91 | 0.91 | 0.20 | 0.97 |
| diagnosis_2 | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) |
In the second the estimate is drawn just once. The SD of the estimate is lower. But the RMSE is not very different.
When redesigning with arguments that are vectors, use list() in redesign, with each list item representing a design you wish to create
A parameter has to be called correctly. And you get no warning if you misname.
why not 200?
A parameter has to be called explicitly
N <- 100
my_N <- function(n = N) n
simplest_design_N2 <-
declare_model(N = my_N(), Y = rnorm(N)) +
declare_inquiry(Q = 0) +
declare_estimator(Y ~ 1)
simplest_design_N2 |> redesign(N = 200) |> draw_data() |> nrow()[1] 100
why not 200?
A parameter has to be called explicitly
N <- 100
my_N <- function(n = N) n
simplest_design_N2 <-
declare_model(N = my_N(N), Y = rnorm(N)) +
declare_inquiry(Q = 0) +
declare_estimator(Y ~ 1)
simplest_design_N2 |> redesign(N = 200) |> draw_data() |> nrow()[1] 200
OK
Here is an example of redesigning where the “parameter” is a function
DeclareDesignA design with hierarchical data and different assignment schemes.
design <-
declare_model(
school = add_level(N = 16,
u_school = rnorm(N, mean = 0)),
classroom = add_level(N = 4,
u_classroom = rnorm(N, mean = 0)),
student = add_level(N = 20,
u_student = rnorm(N, mean = 0))
) +
declare_model(
potential_outcomes(Y ~ .1*Z + u_classroom + u_student + u_school)
) +
declare_assignment(Z = simple_ra(N)) +
declare_measurement(Y = reveal_outcomes(Y ~ Z)) +
declare_inquiry(ATE = mean(Y_Z_1 - Y_Z_0)) +
declare_estimator(Y ~ Z, .method = difference_in_means) Here are the first couple of rows and columns of the resulting data frame.
| school | u_school | classroom | u_classroom | student | u_student | Y_Z_0 | Y_Z_1 | Z | Y |
|---|---|---|---|---|---|---|---|---|---|
| 01 | -1.4 | 01 | -0.2 | 0001 | -0.97 | -2.57 | -2.47 | 0 | -2.57 |
| 01 | -1.4 | 01 | -0.2 | 0002 | 0.67 | -0.93 | -0.83 | 1 | -0.83 |
| 01 | -1.4 | 01 | -0.2 | 0003 | -1.97 | -3.57 | -3.47 | 1 | -3.47 |
| 01 | -1.4 | 01 | -0.2 | 0004 | 0.52 | -1.08 | -0.98 | 0 | -1.08 |
| 01 | -1.4 | 01 | -0.2 | 0005 | 0.60 | -1.00 | -0.90 | 1 | -0.90 |
| 01 | -1.4 | 01 | -0.2 | 0006 | -0.41 | -2.01 | -1.91 | 0 | -2.01 |
Here is the distribution between treatment and control:
We can draw a new set of data and look at the number of subjects in the treatment and control groups.
But what if all students in a given class have to be assigned the same treatment?
assignment_clustered <-
declare_assignment(Z = cluster_ra(clusters = classroom))
estimator_clustered <-
declare_estimator(Y ~ Z, clusters = classroom,
.method = difference_in_means)
design_clustered <-
design |>
replace_step("assignment", assignment_clustered) |>
replace_step("estimator", estimator_clustered)assignment_clustered_blocked <-
declare_assignment(Z = block_and_cluster_ra(blocks = school,
clusters = classroom))
estimator_clustered_blocked <-
declare_estimator(Y ~ Z, blocks = school, clusters = classroom,
.method = difference_in_means)
design_clustered_blocked <-
design |>
replace_step("assignment", assignment_clustered_blocked) |>
replace_step("estimator", estimator_clustered_blocked)| Design | Power | Coverage |
|---|---|---|
| simple | 0.16 | 0.95 |
| (0.01) | (0.01) | |
| complete | 0.20 | 0.96 |
| (0.01) | (0.01) | |
| blocked | 0.42 | 0.95 |
| (0.01) | (0.01) | |
| clustered | 0.06 | 0.96 |
| (0.01) | (0.01) | |
| clustered_blocked | 0.08 | 0.96 |
| (0.01) | (0.01) |
In many designs you seek to assign an integer number of subjects to treatment from some set.
Sometimes however your assignment targets are not integers.
Example:
Two strategies:
Can also be used to set targets
# remotes::install_github("macartan/probra")
library(probra)
set.seed(1)
fabricate(N = 4, size = c(47, 53, 87, 25), n_treated = prob_ra(.5*size)) %>%
janitor::adorn_totals("row") |>
kable(caption = "Setting targets to get 50% targets with minimal variance")| ID | size | n_treated |
|---|---|---|
| 1 | 47 | 23 |
| 2 | 53 | 27 |
| 3 | 87 | 43 |
| 4 | 25 | 13 |
| Total | 212 | 106 |
Can also be used to set for complete assignment with heterogeneous propensities
[1] 0.5
probs <- c(.8, .2)
design <-
declare_model(N = 2,
Y_Z_1 = c(1, 1),
Y_Z_0 = c(-1, 1)) +
declare_inquiry(ATE = 1) +
declare_assignment(
Z = prob_ra(prob = probs),
condition_prs = probs,
Y = reveal_outcomes(Y ~ Z)) +
declare_estimator(Y ~ Z, label = "ht",
.method = horvitz_thompson,
condition_prs = condition_prs)| Inquiry | Estimator | N Sims | Bias | RMSE |
|---|---|---|---|---|
| ATE | ht | 10000 | -0.00 | 2.00 |
| (0.02) | (0.02) |
Unbiased but very very noisy (simulations also noisy)
Indirect control
Indirect assignments are generally generated by applying a direct assignment and then figuring our an implied indirect assignment
Looks better: but there are trade offs between the direct and indirect distributions
Figuring out the optimal procedure requires full diagnosis
Load up:
| \(T2=0\) | \(T2=1\) | |
|---|---|---|
| T1 = 0 | \(50\%\) | \(0\%\) |
| T1 = 1 | \(50\%\) | \(0\%\) |
Spread out:
| \(T2=0\) | \(T2=1\) | |
|---|---|---|
| T1 = 0 | \(25\%\) | \(25\%\) |
| T1 = 1 | \(25\%\) | \(25\%\) |
Three arm it?:
| \(T2=0\) | \(T2=1\) | |
|---|---|---|
| T1 = 0 | \(33.3\%\) | \(33.3\%\) |
| T1 = 1 | \(33.3\%\) | \(0\%\) |
Bunch it?:
| \(T2=0\) | \(T2=1\) | |
|---|---|---|
| T1 = 0 | \(40\%\) | \(20\%\) |
| T1 = 1 | \(20\%\) | \(20\%\) |
Two ways to do factorial assignments in DeclareDesign:
In practice if you have a lot of treatments it can be hard to do full factorial designs – there may be too many combinations.
In such cases people use fractional factorial designs, like the one below (5 treatments but only 8 units!)
| Variation | T1 | T2 | T3 | T4 | T5 |
|---|---|---|---|---|---|
| 1 | 0 | 0 | 0 | 1 | 1 |
| 2 | 0 | 0 | 1 | 0 | 0 |
| 3 | 0 | 1 | 0 | 0 | 1 |
| 4 | 0 | 1 | 1 | 1 | 0 |
| 5 | 1 | 0 | 0 | 1 | 0 |
| 6 | 1 | 0 | 1 | 0 | 1 |
| 7 | 1 | 1 | 0 | 0 | 0 |
| 8 | 1 | 1 | 1 | 1 | 1 |
Then randomly assign units to rows. Note columns might also be blocking covariates.
In R, look at library(survey)
| Unit | T1 | T2 | T3 | T4 | T5 |
|---|---|---|---|---|---|
| 1 | 0 | 0 | 0 | 1 | 1 |
| 2 | 0 | 0 | 1 | 0 | 0 |
| 3 | 0 | 1 | 0 | 0 | 1 |
| 4 | 0 | 1 | 1 | 1 | 0 |
| 5 | 1 | 0 | 0 | 1 | 0 |
| 6 | 1 | 0 | 1 | 0 | 1 |
| 7 | 1 | 1 | 0 | 0 | 0 |
| 8 | 1 | 1 | 1 | 1 | 1 |
library(survey)A focus on power
In the classical approach to testing a hypothesis we ask:
How likely are we to see data like this if indeed the hypothesis is true?
How unlikely is “not very likely”?
When we test a hypothesis we decide first on what sort of evidence we need to see in order to decide that the hypothesis is not reliable.
Othello has a hypothesis that Desdemona is innocent.
Iago confronts him with evidence:
Note that Othello is focused on the probability of the events if she were innocent but not the probability of the events if Iago were trying to trick him.
He is not assessing his belief in whether she is faithful, but rather how likely the data would be if she were faithful.
So:
Illustrating \(p\) values via “randomization inference”
Say you randomized assignment to treatment and your data looked like this.
| Unit | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Treatment | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 |
| Health score | 4 | 2 | 3 | 1 | 2 | 3 | 4 | 8 | 7 | 6 |
Then:
Power is just the probability of getting a significant result rejecting a hypothesis.
Simple enough but it presupposes:
I want to test the hypothesis that a six never comes up on this dice.
Here’s my test:
What is the power of this test?
I want to test the hypothesis that a six never comes up on this dice.
Here’s my test:
What is the power of this test?
Power sometimes seems more complicated because hypothesis rejection involves a calculated probability and so you need the probability of a probability.
I want to test the hypothesis that this dice is fair.
Here’s my test:
Now:
For this we need to figure a rule for rejection. This is based on identifying events that should be unlikely under the hypothesis.
Here is how many 6’s I would expect if the dice is fair:
I can figure out from this that 143 or fewer is really very few and 190 or more is really very many:
Now we need to stipulate some belief about how the world really works—this is not the null hypothesis that we plan to reject, but something that we actually take to be true.
For instance: we think that in fact sixes appear 20% of the time.
Now what’s the probability of seeing at least 190 sixes?
So given I think 6s appear 20% of the time, I think it likely I’ll see at least 190 sixes and reject the hypothesis of a fair dice.
Is arbitrarily flexible
| sim_ID | estimate | p.value |
|---|---|---|
| 1 | 0.81 | 0.00 |
| 2 | 0.40 | 0.04 |
| 3 | 0.88 | 0.00 |
| 4 | 0.72 | 0.00 |
| 5 | 0.38 | 0.05 |
| 6 | 0.44 | 0.02 |
Obviously related to the estimates you might get
A valid \(p\)-value satisfies \(\Pr(p≤x)≤x\) for every \(x \in[0,1]\) (under the null)
| Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|
| 0.50 | 0.00 | 0.20 | 0.20 | 0.70 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) |
| b | Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|---|
| 0 | -0.00 | -0.00 | 0.20 | 0.20 | 0.05 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | |
| 0.25 | 0.25 | -0.00 | 0.20 | 0.20 | 0.23 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | |
| 0.5 | 0.50 | 0.00 | 0.20 | 0.20 | 0.70 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | |
| 1 | 1.00 | 0.00 | 0.20 | 0.20 | 1.00 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.00) |
coming up:
We often focus on sample sizes
But
Power also depends on
Say we have access to a “pre” measure of outcome Y_now; call it Y_base. Y_base is informative about potential outcomes. We are considering using Y_now - Y_base as the outcome instead of Y_now.
N <- 100
rho <- .5
design <-
declare_model(N,
Y_base = rnorm(N),
Y_Z_0 = 1 + correlate(rnorm, given = Y_base, rho = rho),
Y_Z_1 = correlate(rnorm, given = Y_base, rho = rho),
Z = complete_ra(N),
Y_now = Z*Y_Z_1 + (1-Z)*Y_Z_0,
Y_change = Y_now - Y_base) +
declare_inquiry(ATE = mean(Y_Z_1 - Y_Z_0)) +
declare_estimator(Y_now ~ Z, label = "level") +
declare_estimator(Y_change ~ Z, label = "change")+
declare_estimator(Y_now ~ Z + Y_base, label = "RHS")Punchline:
You can see from the null design that power is great but bias is terrible and coverage is way off.
| Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|
| 1.59 | 1.59 | 0.12 | 1.60 | 1.00 | 0.00 |
| (0.01) | (0.01) | (0.00) | (0.01) | (0.00) | (0.00) |
Power without unbiasedness corrupts, absolutely
another_bad_design <-
declare_model(
N = 100,
female = rep(0:1, N/2),
U = rnorm(N),
potential_outcomes(Y ~ female * Z + U)) +
declare_assignment(
Z = block_ra(blocks = female, block_prob = c(.1, .5)),
Y = reveal_outcomes(Y ~ Z)) +
declare_inquiry(ate = mean(Y_Z_1 - Y_Z_0)) +
declare_estimator(Y ~ Z + female, inquiry = "ate",
.method = lm_robust)You can see from the null design that power is great but bias is terrible and coverage is way off.
| Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|
| 0.76 | 0.26 | 0.24 | 0.35 | 0.84 | 0.85 |
| (0.01) | (0.01) | (0.01) | (0.01) | (0.01) | (0.02) |
clustered_design <-
declare_model(
cluster = add_level(N = 10, cluster_shock = rnorm(N)),
individual = add_level(
N = 100,
Y_Z_0 = rnorm(N) + cluster_shock,
Y_Z_1 = rnorm(N) + cluster_shock)) +
declare_inquiry(ATE = mean(Y_Z_1 - Y_Z_0)) +
declare_assignment(Z = cluster_ra(clusters = cluster)) +
declare_measurement(Y = reveal_outcomes(Y ~ Z)) +
declare_estimator(Y ~ Z, inquiry = "ATE")| Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|
| -0.00 | -0.00 | 0.64 | 0.64 | 0.79 | 0.20 |
| (0.01) | (0.01) | (0.01) | (0.01) | (0.01) | (0.01) |
What alerts you to a problem?
| Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|
| 0.00 | -0.00 | 0.66 | 0.65 | 0.06 | 0.94 |
| (0.02) | (0.02) | (0.01) | (0.01) | (0.01) | (0.01) |
design_uncertain <-
declare_model(N = 1000, b = 1+rnorm(1), Y_Z_1 = rnorm(N), Y_Z_2 = rnorm(N) + b, Y_Z_3 = rnorm(N) + b) +
declare_assignment(Z = complete_ra(N = N, num_arms = 3, conditions = 1:3)) +
declare_measurement(Y = reveal_outcomes(Y ~ Z)) +
declare_inquiry(ate = mean(b)) +
declare_estimator(Y ~ factor(Z), term = TRUE)
draw_estimands(design_uncertain) inquiry estimand
1 ate -0.3967765
inquiry estimand
1 ate 0.7887188
Say I run two tests and want to correct for multiple comparisons.
Two approaches. First, by hand:
b = .2
design_mc <-
declare_model(N = 1000, Y_Z_1 = rnorm(N), Y_Z_2 = rnorm(N) + b, Y_Z_3 = rnorm(N) + b) +
declare_assignment(Z = complete_ra(N = N, num_arms = 3, conditions = 1:3)) +
declare_measurement(Y = reveal_outcomes(Y ~ Z)) +
declare_inquiry(ate = b) +
declare_estimator(Y ~ factor(Z), term = TRUE)design_mc |>
simulate_designs(sims = 1000) |>
filter(term != "(Intercept)") |>
group_by(sim_ID) |>
mutate(p_bonferroni = p.adjust(p = p.value, method = "bonferroni"),
p_holm = p.adjust(p = p.value, method = "holm"),
p_fdr = p.adjust(p = p.value, method = "fdr")) |>
ungroup() |>
summarize(
"Power using naive p-values" = mean(p.value <= 0.05),
"Power using Bonferroni correction" = mean(p_bonferroni <= 0.05),
"Power using Holm correction" = mean(p_holm <= 0.05),
"Power using FDR correction" = mean(p_fdr <= 0.05)
) | Power using naive p-values | Power using Bonferroni correction | Power using Holm correction | Power using FDR correction |
|---|---|---|---|
| 0.7374 | 0.6318 | 0.6886 | 0.7032 |
The alternative approach (generally better!) is to design with a custom estimator that includes your corrections.
my_estimator <- function(data)
lm_robust(Y ~ factor(Z), data = data) |>
tidy() |>
filter(term != "(Intercept)") |>
mutate(p.naive = p.value,
p.value = p.adjust(p = p.naive, method = "bonferroni"))
design_mc_2 <- design_mc |>
replace_step(5, declare_estimator(handler = label_estimator(my_estimator)))
run_design(design_mc_2) |>
select(term, estimate, p.value, p.naive) |> kable()| term | estimate | p.value | p.naive |
|---|---|---|---|
| factor(Z)2 | 0.1182516 | 0.2502156 | 0.1251078 |
| factor(Z)3 | 0.1057031 | 0.3337476 | 0.1668738 |
Lets try same thing for a null model (using redesign(design_mc_2, b = 0))
…and power:
| Mean Estimate | Bias | SD Estimate | RMSE | Power | Coverage |
|---|---|---|---|---|---|
| 0.00 | 0.00 | 0.08 | 0.08 | 0.02 | 0.95 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.01) |
| -0.00 | -0.00 | 0.08 | 0.08 | 0.02 | 0.96 |
| (0.00) | (0.00) | (0.00) | (0.00) | (0.00) | (0.01) |
bothered?
Introduction to observational strategies using DeclareDesign
Sometimes you give a medicine but only a nonrandom sample of people actually try to use it. Can you still estimate the medicine’s effect?
| X=0 | X=1 | |
|---|---|---|
| Z=0 | \(\overline{y}_{00}\) (\(n_{00}\)) | \(\overline{y}_{01}\) (\(n_{01}\)) |
| Z=1 | \(\overline{y}_{10}\) (\(n_{10}\)) | \(\overline{y}_{11}\) (\(n_{11}\)) |
Say that people are one of 3 types:
Sometimes you give a medicine but only a non random sample of people actually try to use it. Can you still estimate the medicine’s effect?
| X=0 | X=1 | |
|---|---|---|
| Z=0 | \(\overline{y}_{00}\) (\(n_{00}\)) | \(\overline{y}_{01}\) (\(n_{01}\)) |
| Z=1 | \(\overline{y}_{10}\) (\(n_{10}\)) | \(\overline{y}_{11}\) (\(n_{11}\)) |
We can figure something about types:
| \(X=0\) | \(X=1\) | |
|---|---|---|
| \(Z=0\) | \(\frac{\frac{1}{2}n_c}{\frac{1}{2}n_c + \frac{1}{2}n_n} \overline{y}^0_{c}+\frac{\frac{1}{2}n_n}{\frac{1}{2}n_c + \frac{1}{2}n_n} \overline{y}_{n}\) | \(\overline{y}_{a}\) |
| \(Z=1\) | \(\overline{y}_{n}\) | \(\frac{\frac{1}{2}n_c}{\frac{1}{2}n_c + \frac{1}{2}n_a} \overline{y}^1_{c}+\frac{\frac{1}{2}n_a}{\frac{1}{2}n_c + \frac{1}{2}n_a} \overline{y}_{a}\) |
You give a medicine to 50% but only a non random sample of people actually try to use it. Can you still estimate the medicine’s effect?
| \(X=0\) | \(X=1\) | |
|---|---|---|
| \(Z=0\) | \(\frac{n_c}{n_c + n_n} \overline{y}^0_{c}+\frac{n_n}{n_c + n_n} \overline{y}_n\) | \(\overline{y}_{a}\) |
| (n) | (\(\frac{1}{2}(n_c + n_n)\)) | (\(\frac{1}{2}n_a\)) |
| \(Z=1\) | \(\overline{y}_{n}\) | \(\frac{n_c}{n_c + n_a} \overline{y}^1_{c}+\frac{n_a}{n_c + n_a} \overline{y}_{a}\) |
| (n) | (\(\frac{1}{2}n_n\)) | (\(\frac{1}{2}(n_a+n_c)\)) |
Key insight: the contributions of the \(a\)s and \(n\)s are the same in the \(Z=0\) and \(Z=1\) groups so if you difference you are left with the changes in the contributions of the \(c\)s.
Average in \(Z=0\) group: \(\frac{{n_c} \overline{y}^0_{c}+ \left(n_{n}\overline{y}_{n} +{n_a} \overline{y}_a\right)}{n_a+n_c+n_n}\)
Average in \(Z=1\) group: \(\frac{{n_c} \overline{y}^1_{c} + \left(n_{n}\overline{y}_{n} +{n_a} \overline{y}_a \right)}{n_a+n_c+n_n}\)
So, the difference is the ITT: \(({\overline{y}^1_c-\overline{y}^0_c})\frac{n_c}{n}\)
Last step:
\[ITT = ({\overline{y}^1_c-\overline{y}^0_c})\frac{n_c}{n}\]
\[\leftrightarrow\]
\[LATE = \frac{ITT}{\frac{n_c}{n}}= \frac{\text{Intent to treat effect}}{\text{First stage effect}}\]
declaration_iv <-
declare_model(
N = 100,
U = rnorm(N),
potential_outcomes(D ~ if_else(Z + U > 0, 1, 0),
conditions = list(Z = c(0, 1))),
potential_outcomes(Y ~ 0.1 * D + 0.25 + U,
conditions = list(D = c(0, 1))),
complier = D_Z_1 == 1 & D_Z_0 == 0
) +
declare_inquiry(ATE = mean(Y_D_1 - Y_D_0),
LATE = mean(Y_D_1[complier] - Y_D_0[complier])) +
declare_assignment(Z = complete_ra(N, prob = 0.5)) +
declare_measurement(D = reveal_outcomes(D ~ Z),
Y = reveal_outcomes(Y ~ D)) +
declare_estimator(Y ~ D, inquiry = "ATE", label = "OLS") +
declare_estimator(Y ~ D | Z, .method = iv_robust, inquiry = "LATE",
label = "IV") | Inquiry | Estimator | Mean Estimand | Mean Estimate | Bias | RMSE |
|---|---|---|---|---|---|
| ATE | OLS | 0.10 | 1.55 | 1.45 | 1.46 |
| (0.00) | (0.00) | (0.00) | (0.00) | ||
| LATE | IV | 0.10 | -0.05 | -0.15 | 0.80 |
| (0.00) | (0.01) | (0.01) | (0.03) |
Note:
Key idea: the evolution of units in the control group allow you to impute what the evolution of units in the treatment group would have been had they not been treated
We have group \(A\) that enters treatment at some point and group \(B\) that never does
The estimate:
\[\hat\tau = (\mathbb{E}[Y^A | post] - \mathbb{E}[Y^A | pre]) -(\mathbb{E}[Y^B | post] - \mathbb{E}[Y^B | pre])\] (how different is the change in \(A\) compared to the change in \(B\)?)
can be written:
\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre])\]
Cleaning up
\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre])\]
\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | post]) + ((\mathbb{E}[Y_0^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre]))\]
\[\hat\tau_{ATT} = \tau_{ATT} + \text{Difference in trends}\]
n_units <- 2
design <-
declare_model(
unit = add_level(N = n_units, I = 1:N),
time = add_level(N = 6, T = 1:N, nest = FALSE),
obs = cross_levels(by = join_using(unit, time))) +
declare_model(potential_outcomes(Y ~ I + T^.5 + Z*T)) +
declare_assignment(Z = 1*(I>(n_units/2))*(T>3)) +
declare_measurement(Y = reveal_outcomes(Y~Z)) +
declare_inquiry(ATE = mean(Y_Z_1 - Y_Z_0),
ATT = mean(Y_Z_1[Z==1] - Y_Z_0[Z==1])) +
declare_estimator(Y ~ Z, label = "naive") +
declare_estimator(Y ~ Z + I, label = "FE1") +
declare_estimator(Y ~ Z + as.factor(T), label = "FE2") +
declare_estimator(Y ~ Z + I + as.factor(T), label = "FE3") | unit | I | time | T | obs | Y_Z_0 | Y_Z_1 | Z | Y |
|---|---|---|---|---|---|---|---|---|
| 1 | 1 | 1 | 1 | 01 | 2.000000 | 3.000000 | 0 | 2.000000 |
| 2 | 2 | 1 | 1 | 02 | 3.000000 | 4.000000 | 0 | 3.000000 |
| 1 | 1 | 2 | 2 | 03 | 2.414214 | 4.414214 | 0 | 2.414214 |
| 2 | 2 | 2 | 2 | 04 | 3.414214 | 5.414214 | 0 | 3.414214 |
| 1 | 1 | 3 | 3 | 05 | 2.732051 | 5.732051 | 0 | 2.732051 |
| 2 | 2 | 3 | 3 | 06 | 3.732051 | 6.732051 | 0 | 3.732051 |
Here only the two way fixed effects is unbiased and only for the ATT.
The ATT here is averaging over effects for treated units (later periods only). We know nothing about the size of effects in earlier periods when all units are in control!
| Inquiry | Estimator | Bias |
|---|---|---|
| ATE | FE1 | 2.25 |
| ATE | FE2 | 6.50 |
| ATE | FE3 | 1.50 |
| ATE | naive | 5.40 |
| ATT | FE1 | 0.75 |
| ATT | FE2 | 5.00 |
| ATT | FE3 | 0.00 |
| ATT | naive | 3.90 |
| Inquiry | Estimator | Bias |
|---|---|---|
| ATE | FE1 | 2.25 |
| ATE | FE2 | 6.50 |
| ATE | FE3 | 1.50 |
| ATE | naive | 5.40 |
| ATT | FE1 | 0.75 |
| ATT | FE2 | 5.00 |
| ATT | FE3 | 0.00 |
| ATT | naive | 3.90 |
Things get much more complicated when there is (a) heterogeneous timing in treatment take up and (b) heterogeneous effects
It’s only recently been appreciated how tricky things can get
But we already have an intuition from our analysis of trials with heterogeneous assignment and heterogeneous effects:
in such cases fixed effects analysis weights stratum level treatment effects by the variance in assignment to treatment
something similar here
Just two units assigned at different times:
trend = 0
design <-
declare_model(
unit = add_level(N = 2, ui = rnorm(N), I = 1:N),
time = add_level(N = 6, ut = rnorm(N), T = 1:N, nest = FALSE),
obs = cross_levels(by = join_using(unit, time))) +
declare_model(
potential_outcomes(Y ~ trend*T + (1+Z)*(I == 2))) +
declare_assignment(Z = 1*((I == 1) * (T>3) + (I == 2) * (T>5))) +
declare_measurement(Y = reveal_outcomes(Y~Z),
I_c = I - mean(I)) +
declare_inquiry(mean(Y_Z_1 - Y_Z_0)) +
declare_estimator(Y ~ Z, label = "1. naive") +
declare_estimator(Y ~ Z + I, label = "2. FE1") +
declare_estimator(Y ~ Z + as.factor(T), label = "3. FE2") +
declare_estimator(Y ~ Z + I + as.factor(T), label = "4. FE3") +
declare_estimator(Y ~ Z*I_c + as.factor(T), label = "5. Sat") | Estimator | Mean Estimand | Mean Estimate |
|---|---|---|
| 1. naive | 0.50 | -0.12 |
| (0.00) | (0.00) | |
| 2. FE1 | 0.50 | 0.36 |
| (0.00) | (0.00) | |
| 3. FE2 | 0.50 | -1.00 |
| (0.00) | (0.00) | |
| 4. FE3 | 0.50 | 0.25 |
| (0.00) | (0.00) | |
| 5. Sat | 0.50 | 0.50 |
| (0.00) | (0.00) |
See causal infernece slides for intuitions on what is happening here.
Errors and diagnostics
See excellent introduction: Lee and Lemieux (2010)
Kids born on 31 August start school a year younger than kids born on 1 September: does starting younger help or hurt?
Kids born on 12 September 1983 are more likely to register Republican than kids born on 10 September 1983: can this identify the effects of registration on long term voting?
A district in which Republicans got 50.1% of the vote get a Republican representative while districts in which Republicans got 49.9% of the vote do not: does having a Republican representative make a difference for these districts?
Setting:
Two arguments:
Continuity: \(\mathbb{E}[Y(1)|X=x]\) and \(\mathbb{E}[Y(0)|X=x]\) are continuous (at \(x=0\)) in \(x\): so \(\lim_{\hat x \rightarrow 0}\mathbb{E}[Y(0)|X=\hat x] = \mathbb{E}[Y(0)|X=\hat 0]\)
Local randomization: tiny things that determine exact values of \(x\) are as if random and so we can think of a local experiment around \(X=0\).
Note:
Exclusion restriction is implicit in continuity: If something else happens at the threshold then the conditional expectation functions jump at the thresholds
Implicit: \(X\) is exogenous in the sense that units cannot adjust \(X\) in order to be on one or the other side of the threshold
Typically researchers show:
Typically researchers show:
In addition:
Sometimes:
library(rdss) # for helper functions
library(rdrobust)
cutoff <- 0.5
bandwidth <- 0.5
control <- function(X) {
as.vector(poly(X, 4, raw = TRUE) %*% c(.7, -.8, .5, 1))}
treatment <- function(X) {
as.vector(poly(X, 4, raw = TRUE) %*% c(0, -1.5, .5, .8)) + .25}
rdd_design <-
declare_model(
N = 1000,
U = rnorm(N, 0, 0.1),
X = runif(N, 0, 1) + U - cutoff,
D = 1 * (X > 0),
Y_D_0 = control(X) + U,
Y_D_1 = treatment(X) + U
) +
declare_inquiry(LATE = treatment(0) - control(0)) +
declare_measurement(Y = reveal_outcomes(Y ~ D)) +
declare_sampling(S = X > -bandwidth & X < bandwidth) +
declare_estimator(Y ~ D*X, term = "D", label = "lm") +
declare_estimator(
Y, X,
term = "Bias-Corrected",
.method = rdrobust_helper,
label = "optimal"
)Note rdrobust implements:
See Calonico, Cattaneo, and Titiunik (2014) and related papers ? rdrobust::rdrobust
| Estimator | Mean Estimate | Bias | SD Estimate | Coverage |
|---|---|---|---|---|
| lm | 0.23 | -0.02 | 0.01 | 0.64 |
| (0.00) | (0.00) | (0.00) | (0.02) | |
| optimal | 0.25 | 0.00 | 0.03 | 0.89 |
| (0.00) | (0.00) | (0.00) | (0.01) |